2024-09-16T13:05:02.281 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T13:05:02.297 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T13:05:02.373 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_cache_writeback} email: sangadi@redhat.com first_in_suite: false job_id: '7907295' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: rbd cache: true rbd cache policy: writeback global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - mons down - mon down - out of quorum sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_sangadi@teuthology priority: 130 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9764 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - rbd_fsx: clients: - client.0 ops: 6000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_10:44:09 tube: smithi user: sangadi verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T13:05:02.374 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-16T13:05:02.374 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-16T13:05:02.374 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T13:05:02.376 INFO:teuthology.task.internal:Checking packages... 2024-09-16T13:05:02.398 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-16T13:05:02.398 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T13:05:02.398 INFO:teuthology.packaging:ref: None 2024-09-16T13:05:02.398 INFO:teuthology.packaging:tag: None 2024-09-16T13:05:02.398 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T13:05:02.398 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-16T13:05:02.398 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T13:05:02.547 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643-gd75f2382-1focal 2024-09-16T13:05:02.549 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T13:05:02.556 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T13:05:02.556 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T13:05:02.564 INFO:teuthology.task.internal:Saving configuration 2024-09-16T13:05:02.576 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T13:05:02.583 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T13:05:02.604 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 12:58:50.366893', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T13:05:02.623 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 12:58:50.365450', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T13:05:02.624 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T13:05:02.630 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-09-16T13:05:02.630 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-16T13:05:02.630 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T13:05:02.689 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f947bbebac0>, signals=[15]) 2024-09-16T13:05:02.689 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T13:05:02.695 INFO:teuthology.task.internal:Opening connections... 2024-09-16T13:05:02.695 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:05:02.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:05:02.788 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:05:02.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:05:02.877 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T13:05:02.884 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-09-16T13:05:02.891 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-09-16T13:05:02.891 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-09-16T13:05:02.939 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="Ubuntu" 2024-09-16T13:05:02.939 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T13:05:02.939 INFO:teuthology.orchestra.run.smithi098.stdout:ID=ubuntu 2024-09-16T13:05:02.939 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE=debian 2024-09-16T13:05:02.939 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T13:05:02.939 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="20.04" 2024-09-16T13:05:02.939 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T13:05:02.940 INFO:teuthology.orchestra.run.smithi098.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T13:05:02.940 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T13:05:02.940 INFO:teuthology.orchestra.run.smithi098.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T13:05:02.940 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_CODENAME=focal 2024-09-16T13:05:02.940 INFO:teuthology.orchestra.run.smithi098.stdout:UBUNTU_CODENAME=focal 2024-09-16T13:05:02.940 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-09-16T13:05:02.960 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-09-16T13:05:02.967 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-09-16T13:05:02.967 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-09-16T13:05:03.013 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="Ubuntu" 2024-09-16T13:05:03.013 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T13:05:03.014 INFO:teuthology.orchestra.run.smithi151.stdout:ID=ubuntu 2024-09-16T13:05:03.014 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE=debian 2024-09-16T13:05:03.014 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T13:05:03.014 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="20.04" 2024-09-16T13:05:03.014 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T13:05:03.014 INFO:teuthology.orchestra.run.smithi151.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T13:05:03.014 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T13:05:03.014 INFO:teuthology.orchestra.run.smithi151.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T13:05:03.014 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_CODENAME=focal 2024-09-16T13:05:03.014 INFO:teuthology.orchestra.run.smithi151.stdout:UBUNTU_CODENAME=focal 2024-09-16T13:05:03.014 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-09-16T13:05:03.035 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T13:05:03.044 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T13:05:03.049 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T13:05:03.049 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-09-16T13:05:03.052 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-09-16T13:05:03.057 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T13:05:03.063 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T13:05:03.064 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-09-16T13:05:03.066 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-09-16T13:05:03.084 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T13:05:03.131 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T13:05:03.145 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T13:05:03.145 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T13:05:03.145 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T13:05:03.146 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T13:05:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:05:03.151 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-16T13:05:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:05:03.152 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-09-16T13:05:03.158 INFO:teuthology.orchestra.run.smithi098.stdout:5.4.0-124-generic 2024-09-16T13:05:03.158 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.4.0-124-generic 2024-09-16T13:05:03.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get clean 2024-09-16T13:05:03.159 INFO:teuthology.orchestra.run.smithi151.stdout:5.4.0-124-generic 2024-09-16T13:05:03.159 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.4.0-124-generic 2024-09-16T13:05:03.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get clean 2024-09-16T13:05:03.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get update 2024-09-16T13:05:03.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-09-16T13:05:03.449 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T13:05:03.462 INFO:teuthology.orchestra.run.smithi151.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T13:05:03.561 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T13:05:03.561 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T13:05:03.643 INFO:teuthology.orchestra.run.smithi151.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T13:05:03.643 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T13:05:03.745 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T13:05:03.761 INFO:teuthology.orchestra.run.smithi151.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T13:05:03.855 INFO:teuthology.orchestra.run.smithi151.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T13:05:03.897 INFO:teuthology.orchestra.run.smithi151.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T13:05:03.900 INFO:teuthology.orchestra.run.smithi151.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T13:05:03.900 INFO:teuthology.orchestra.run.smithi151.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T13:05:03.901 INFO:teuthology.orchestra.run.smithi151.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T13:05:03.907 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T13:05:03.918 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T13:05:03.921 INFO:teuthology.orchestra.run.smithi151.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T13:05:03.922 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T13:05:03.923 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T13:05:03.924 INFO:teuthology.orchestra.run.smithi151.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T13:05:03.924 INFO:teuthology.orchestra.run.smithi151.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T13:05:03.928 INFO:teuthology.orchestra.run.smithi151.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T13:05:03.935 INFO:teuthology.orchestra.run.smithi151.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T13:05:03.937 INFO:teuthology.orchestra.run.smithi151.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T13:05:03.937 INFO:teuthology.orchestra.run.smithi151.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T13:05:03.938 INFO:teuthology.orchestra.run.smithi151.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T13:05:03.938 INFO:teuthology.orchestra.run.smithi151.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T13:05:03.948 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T13:05:03.949 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T13:05:03.949 INFO:teuthology.orchestra.run.smithi151.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T13:05:03.953 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T13:05:03.953 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T13:05:03.969 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T13:05:03.973 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T13:05:03.976 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T13:05:03.976 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T13:05:03.977 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T13:05:03.977 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T13:05:03.978 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T13:05:03.997 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T13:05:03.997 INFO:teuthology.orchestra.run.smithi151.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T13:05:04.136 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T13:05:04.137 INFO:teuthology.orchestra.run.smithi151.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,566 kB] 2024-09-16T13:05:04.380 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,566 kB] 2024-09-16T13:05:04.521 INFO:teuthology.orchestra.run.smithi151.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T13:05:04.550 INFO:teuthology.orchestra.run.smithi151.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T13:05:04.562 INFO:teuthology.orchestra.run.smithi151.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T13:05:04.562 INFO:teuthology.orchestra.run.smithi151.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,240 kB] 2024-09-16T13:05:04.585 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T13:05:04.601 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T13:05:04.602 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,240 kB] 2024-09-16T13:05:04.656 INFO:teuthology.orchestra.run.smithi151.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T13:05:04.656 INFO:teuthology.orchestra.run.smithi151.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [454 kB] 2024-09-16T13:05:04.672 INFO:teuthology.orchestra.run.smithi151.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T13:05:04.672 INFO:teuthology.orchestra.run.smithi151.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T13:05:04.694 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T13:05:04.699 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [454 kB] 2024-09-16T13:05:04.709 INFO:teuthology.orchestra.run.smithi151.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T13:05:04.710 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T13:05:04.710 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T13:05:04.728 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T13:05:04.729 INFO:teuthology.orchestra.run.smithi151.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T13:05:04.737 INFO:teuthology.orchestra.run.smithi151.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T13:05:04.737 INFO:teuthology.orchestra.run.smithi151.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T13:05:04.738 INFO:teuthology.orchestra.run.smithi151.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T13:05:04.766 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T13:05:04.769 INFO:teuthology.orchestra.run.smithi151.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T13:05:04.769 INFO:teuthology.orchestra.run.smithi151.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T13:05:04.770 INFO:teuthology.orchestra.run.smithi151.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T13:05:04.770 INFO:teuthology.orchestra.run.smithi151.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T13:05:04.775 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T13:05:04.781 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T13:05:04.781 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T13:05:04.830 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T13:05:04.831 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T13:05:04.831 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T13:05:04.832 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T13:05:04.833 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T13:05:04.852 INFO:teuthology.orchestra.run.smithi151.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T13:05:04.852 INFO:teuthology.orchestra.run.smithi151.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T13:05:04.852 INFO:teuthology.orchestra.run.smithi151.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T13:05:04.852 INFO:teuthology.orchestra.run.smithi151.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T13:05:04.912 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T13:05:04.913 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T13:05:04.913 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T13:05:08.361 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 21.9 MB in 3s (7,289 kB/s) 2024-09-16T13:05:08.429 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 21.9 MB in 3s (7,270 kB/s) 2024-09-16T13:05:09.872 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2024-09-16T13:05:09.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T13:05:09.971 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2024-09-16T13:05:10.001 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-16T13:05:10.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T13:05:10.099 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-16T13:05:10.166 INFO:teuthology.orchestra.run.smithi151.stdout:Building dependency tree... 2024-09-16T13:05:10.167 INFO:teuthology.orchestra.run.smithi151.stdout:Reading state information... 2024-09-16T13:05:10.292 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-09-16T13:05:10.293 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-09-16T13:05:10.361 INFO:teuthology.orchestra.run.smithi151.stdout:The following additional packages will be installed: 2024-09-16T13:05:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T13:05:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T13:05:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T13:05:10.363 INFO:teuthology.orchestra.run.smithi151.stdout:Suggested packages: 2024-09-16T13:05:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T13:05:10.363 INFO:teuthology.orchestra.run.smithi151.stdout:Recommended packages: 2024-09-16T13:05:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: thermald 2024-09-16T13:05:10.429 INFO:teuthology.orchestra.run.smithi151.stdout:The following NEW packages will be installed: 2024-09-16T13:05:10.429 INFO:teuthology.orchestra.run.smithi151.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T13:05:10.430 INFO:teuthology.orchestra.run.smithi151.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T13:05:10.430 INFO:teuthology.orchestra.run.smithi151.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T13:05:10.431 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages will be upgraded: 2024-09-16T13:05:10.431 INFO:teuthology.orchestra.run.smithi151.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T13:05:10.492 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-09-16T13:05:10.492 INFO:teuthology.orchestra.run.smithi098.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T13:05:10.492 INFO:teuthology.orchestra.run.smithi098.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T13:05:10.492 INFO:teuthology.orchestra.run.smithi098.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T13:05:10.493 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-09-16T13:05:10.493 INFO:teuthology.orchestra.run.smithi098.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T13:05:10.493 INFO:teuthology.orchestra.run.smithi098.stdout:Recommended packages: 2024-09-16T13:05:10.493 INFO:teuthology.orchestra.run.smithi098.stdout: thermald 2024-09-16T13:05:10.552 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-09-16T13:05:10.552 INFO:teuthology.orchestra.run.smithi098.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T13:05:10.552 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T13:05:10.553 INFO:teuthology.orchestra.run.smithi098.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T13:05:10.554 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages will be upgraded: 2024-09-16T13:05:10.555 INFO:teuthology.orchestra.run.smithi098.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T13:05:10.610 INFO:teuthology.orchestra.run.smithi098.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T13:05:10.610 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 77.2 MB of archives. 2024-09-16T13:05:10.610 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T13:05:10.610 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T13:05:10.613 INFO:teuthology.orchestra.run.smithi151.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T13:05:10.613 INFO:teuthology.orchestra.run.smithi151.stdout:Need to get 77.2 MB of archives. 2024-09-16T13:05:10.613 INFO:teuthology.orchestra.run.smithi151.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T13:05:10.613 INFO:teuthology.orchestra.run.smithi151.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T13:05:10.946 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T13:05:11.038 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T13:05:11.566 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T13:05:11.566 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T13:05:11.567 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T13:05:11.661 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T13:05:11.671 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T13:05:11.779 INFO:teuthology.orchestra.run.smithi151.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T13:05:12.052 INFO:teuthology.orchestra.run.smithi151.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T13:05:12.396 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 77.2 MB in 1s (69.5 MB/s) 2024-09-16T13:05:12.606 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T13:05:13.404 INFO:teuthology.orchestra.run.smithi151.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T13:05:13.405 INFO:teuthology.orchestra.run.smithi151.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T13:05:13.405 INFO:teuthology.orchestra.run.smithi151.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T13:05:13.690 INFO:teuthology.orchestra.run.smithi151.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T13:05:13.701 INFO:teuthology.orchestra.run.smithi151.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T13:05:14.320 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T13:05:14.325 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T13:05:14.411 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 77.2 MB in 3s (23.6 MB/s) 2024-09-16T13:05:14.478 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:14.686 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T13:05:16.439 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T13:05:16.456 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T13:05:16.497 INFO:teuthology.orchestra.run.smithi151.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T13:05:16.502 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T13:05:16.579 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:16.652 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:17.758 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T13:05:17.774 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T13:05:17.817 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:18.598 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T13:05:18.615 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T13:05:18.719 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:19.794 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T13:05:19.803 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T13:05:19.844 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:22.770 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T13:05:22.914 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T13:05:23.248 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T13:05:23.368 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T13:05:23.721 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T13:05:23.735 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T13:05:23.796 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T13:05:24.671 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T13:05:24.797 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T13:05:25.152 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T13:05:25.278 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T13:05:25.623 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T13:05:25.637 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T13:05:25.682 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T13:05:29.453 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T13:05:29.473 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T13:05:29.536 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:31.255 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T13:05:31.274 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T13:05:31.330 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:32.332 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T13:05:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T13:05:32.815 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:34.044 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:34.048 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T13:05:34.176 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T13:05:34.535 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:35.200 INFO:teuthology.orchestra.run.smithi098.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T13:05:35.200 INFO:teuthology.orchestra.run.smithi098.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T13:05:35.337 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:36.503 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:36.589 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T13:05:36.715 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T13:05:36.825 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:36.942 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T13:05:37.051 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T13:05:37.203 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T13:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T13:05:37.665 INFO:teuthology.orchestra.run.smithi151.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T13:05:37.665 INFO:teuthology.orchestra.run.smithi151.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T13:05:37.805 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:39.015 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T13:05:39.141 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T13:05:39.293 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:39.419 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T13:05:39.546 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T13:05:39.689 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T13:05:39.761 INFO:teuthology.orchestra.run.smithi151.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T13:05:39.761 INFO:teuthology.orchestra.run.smithi151.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T13:05:59.294 INFO:teuthology.orchestra.run.smithi098.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T13:05:59.294 INFO:teuthology.orchestra.run.smithi098.stdout:Sourcing file `/etc/default/grub' 2024-09-16T13:05:59.317 INFO:teuthology.orchestra.run.smithi098.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T13:05:59.343 INFO:teuthology.orchestra.run.smithi098.stdout:Generating grub configuration file ... 2024-09-16T13:05:59.705 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T13:05:59.730 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T13:06:00.137 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T13:06:00.148 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T13:06:01.801 INFO:teuthology.orchestra.run.smithi151.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T13:06:01.801 INFO:teuthology.orchestra.run.smithi151.stdout:Sourcing file `/etc/default/grub' 2024-09-16T13:06:01.814 INFO:teuthology.orchestra.run.smithi151.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T13:06:01.822 INFO:teuthology.orchestra.run.smithi151.stdout:Generating grub configuration file ... 2024-09-16T13:06:02.194 INFO:teuthology.orchestra.run.smithi151.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T13:06:02.216 INFO:teuthology.orchestra.run.smithi151.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T13:06:02.596 INFO:teuthology.orchestra.run.smithi151.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T13:06:02.608 INFO:teuthology.orchestra.run.smithi151.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T13:06:02.670 INFO:teuthology.orchestra.run.smithi098.stdout:done 2024-09-16T13:06:03.095 DEBUG:teuthology.orchestra.run.smithi098:> dpkg -s linux-image-generic 2024-09-16T13:06:03.160 INFO:teuthology.orchestra.run.smithi098.stdout:Package: linux-image-generic 2024-09-16T13:06:03.160 INFO:teuthology.orchestra.run.smithi098.stdout:Status: install ok installed 2024-09-16T13:06:03.160 INFO:teuthology.orchestra.run.smithi098.stdout:Priority: optional 2024-09-16T13:06:03.160 INFO:teuthology.orchestra.run.smithi098.stdout:Section: kernel 2024-09-16T13:06:03.160 INFO:teuthology.orchestra.run.smithi098.stdout:Installed-Size: 20 2024-09-16T13:06:03.160 INFO:teuthology.orchestra.run.smithi098.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T13:06:03.160 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: amd64 2024-09-16T13:06:03.160 INFO:teuthology.orchestra.run.smithi098.stdout:Source: linux-meta 2024-09-16T13:06:03.161 INFO:teuthology.orchestra.run.smithi098.stdout:Version: 5.4.0.195.193 2024-09-16T13:06:03.161 INFO:teuthology.orchestra.run.smithi098.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T13:06:03.161 INFO:teuthology.orchestra.run.smithi098.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T13:06:03.161 INFO:teuthology.orchestra.run.smithi098.stdout:Recommends: thermald 2024-09-16T13:06:03.161 INFO:teuthology.orchestra.run.smithi098.stdout:Description: Generic Linux kernel image 2024-09-16T13:06:03.161 INFO:teuthology.orchestra.run.smithi098.stdout: This package will always depend on the latest generic kernel image 2024-09-16T13:06:03.161 INFO:teuthology.orchestra.run.smithi098.stdout: available. 2024-09-16T13:06:03.161 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T13:06:03.161 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T13:06:03.161 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T13:06:03.162 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T13:06:03.162 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T13:06:03.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get clean 2024-09-16T13:06:03.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-09-16T13:06:03.379 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T13:06:03.379 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T13:06:03.386 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T13:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T13:06:04.413 INFO:teuthology.orchestra.run.smithi151.stdout:done 2024-09-16T13:06:04.859 DEBUG:teuthology.orchestra.run.smithi151:> dpkg -s linux-image-generic 2024-09-16T13:06:04.897 INFO:teuthology.orchestra.run.smithi151.stdout:Package: linux-image-generic 2024-09-16T13:06:04.897 INFO:teuthology.orchestra.run.smithi151.stdout:Status: install ok installed 2024-09-16T13:06:04.897 INFO:teuthology.orchestra.run.smithi151.stdout:Priority: optional 2024-09-16T13:06:04.897 INFO:teuthology.orchestra.run.smithi151.stdout:Section: kernel 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout:Installed-Size: 20 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: amd64 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout:Source: linux-meta 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout:Version: 5.4.0.195.193 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout:Recommends: thermald 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout:Description: Generic Linux kernel image 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout: This package will always depend on the latest generic kernel image 2024-09-16T13:06:04.898 INFO:teuthology.orchestra.run.smithi151.stdout: available. 2024-09-16T13:06:04.898 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T13:06:04.898 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T13:06:04.898 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T13:06:04.899 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T13:06:04.899 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T13:06:04.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get clean 2024-09-16T13:06:04.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get update 2024-09-16T13:06:05.101 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T13:06:05.102 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T13:06:05.108 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T13:06:05.125 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T13:06:05.264 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-16T13:06:05.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T13:06:05.361 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-16T13:06:05.578 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-09-16T13:06:05.578 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-09-16T13:06:05.816 INFO:teuthology.orchestra.run.smithi098.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T13:06:05.816 INFO:teuthology.orchestra.run.smithi098.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T13:06:05.819 DEBUG:teuthology.orchestra.run.smithi098:> dpkg -s linux-image-generic 2024-09-16T13:06:05.840 INFO:teuthology.orchestra.run.smithi098.stdout:Package: linux-image-generic 2024-09-16T13:06:05.840 INFO:teuthology.orchestra.run.smithi098.stdout:Status: install ok installed 2024-09-16T13:06:05.840 INFO:teuthology.orchestra.run.smithi098.stdout:Priority: optional 2024-09-16T13:06:05.840 INFO:teuthology.orchestra.run.smithi098.stdout:Section: kernel 2024-09-16T13:06:05.840 INFO:teuthology.orchestra.run.smithi098.stdout:Installed-Size: 20 2024-09-16T13:06:05.840 INFO:teuthology.orchestra.run.smithi098.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T13:06:05.840 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: amd64 2024-09-16T13:06:05.840 INFO:teuthology.orchestra.run.smithi098.stdout:Source: linux-meta 2024-09-16T13:06:05.840 INFO:teuthology.orchestra.run.smithi098.stdout:Version: 5.4.0.195.193 2024-09-16T13:06:05.841 INFO:teuthology.orchestra.run.smithi098.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T13:06:05.841 INFO:teuthology.orchestra.run.smithi098.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T13:06:05.841 INFO:teuthology.orchestra.run.smithi098.stdout:Recommends: thermald 2024-09-16T13:06:05.841 INFO:teuthology.orchestra.run.smithi098.stdout:Description: Generic Linux kernel image 2024-09-16T13:06:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: This package will always depend on the latest generic kernel image 2024-09-16T13:06:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: available. 2024-09-16T13:06:05.841 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-16T13:06:05.847 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.QNt3Uiy7DX 2024-09-16T13:06:05.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub/grub.cfg /tmp/tmp.QNt3Uiy7DX 2024-09-16T13:06:05.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.QNt3Uiy7DX 2024-09-16T13:06:05.959 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.QNt3Uiy7DX is 9KB 2024-09-16T13:06:05.975 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.QNt3Uiy7DX 2024-09-16T13:06:05.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T13:06:06.037 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:06:06.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T13:06:06.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T13:06:06.066 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T13:06:06.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo update-grub 2024-09-16T13:06:06.958 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2024-09-16T13:06:06.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T13:06:07.053 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2024-09-16T13:06:07.272 INFO:teuthology.orchestra.run.smithi151.stdout:Building dependency tree... 2024-09-16T13:06:07.272 INFO:teuthology.orchestra.run.smithi151.stdout:Reading state information... 2024-09-16T13:06:07.517 INFO:teuthology.orchestra.run.smithi151.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T13:06:07.517 INFO:teuthology.orchestra.run.smithi151.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T13:06:07.519 DEBUG:teuthology.orchestra.run.smithi151:> dpkg -s linux-image-generic 2024-09-16T13:06:07.541 INFO:teuthology.orchestra.run.smithi151.stdout:Package: linux-image-generic 2024-09-16T13:06:07.541 INFO:teuthology.orchestra.run.smithi151.stdout:Status: install ok installed 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Priority: optional 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Section: kernel 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Installed-Size: 20 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: amd64 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Source: linux-meta 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Version: 5.4.0.195.193 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Recommends: thermald 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout:Description: Generic Linux kernel image 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout: This package will always depend on the latest generic kernel image 2024-09-16T13:06:07.542 INFO:teuthology.orchestra.run.smithi151.stdout: available. 2024-09-16T13:06:07.543 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-09-16T13:06:07.546 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub' 2024-09-16T13:06:07.557 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T13:06:07.567 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-09-16T13:06:07.589 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.433tEezDqR 2024-09-16T13:06:07.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub/grub.cfg /tmp/tmp.433tEezDqR 2024-09-16T13:06:07.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.433tEezDqR 2024-09-16T13:06:07.705 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.433tEezDqR is 9KB 2024-09-16T13:06:07.720 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.433tEezDqR 2024-09-16T13:06:07.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T13:06:07.783 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:06:07.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T13:06:07.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T13:06:07.817 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T13:06:07.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo update-grub 2024-09-16T13:06:07.826 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T13:06:07.838 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T13:06:08.234 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T13:06:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T13:06:09.242 INFO:teuthology.orchestra.run.smithi151.stderr:Sourcing file `/etc/default/grub' 2024-09-16T13:06:09.254 INFO:teuthology.orchestra.run.smithi151.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T13:06:09.264 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-09-16T13:06:09.556 INFO:teuthology.orchestra.run.smithi151.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T13:06:09.567 INFO:teuthology.orchestra.run.smithi151.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T13:06:09.729 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-09-16T13:06:09.730 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-09-16T13:06:10.001 INFO:teuthology.orchestra.run.smithi151.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T13:06:10.013 INFO:teuthology.orchestra.run.smithi151.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T13:06:10.784 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-09-16T13:06:10.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2024-09-16T13:06:39.762 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T13:06:39.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T13:06:39.764 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:06:40.790 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T13:06:40.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-16T13:06:40.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:06:58.237 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-16T13:06:59.261 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-09-16T13:07:07.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T13:07:07.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:07:08.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-16T13:07:08.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:07:10.301 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-16T13:07:11.325 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-09-16T13:07:22.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T13:07:22.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:07:23.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-16T13:07:23.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:07:54.022 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-16T13:07:54.611 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T13:07:54.611 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-16T13:07:54.612 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-16T13:07:54.618 INFO:teuthology.orchestra.run.smithi098.stdout:5.4.0-195-generic 2024-09-16T13:07:54.618 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T13:07:54.618 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T13:07:54.618 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T13:07:54.813 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-09-16T13:07:55.620 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T13:07:55.620 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T13:07:55.714 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-09-16T13:07:55.742 DEBUG:teuthology.parallel:result is None 2024-09-16T13:08:09.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-16T13:08:09.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:08:10.061 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-09-16T13:08:10.641 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-16T13:08:10.641 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-195-generic"... 2024-09-16T13:08:10.641 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-09-16T13:08:10.687 INFO:teuthology.orchestra.run.smithi151.stdout:5.4.0-195-generic 2024-09-16T13:08:10.688 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T13:08:10.688 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T13:08:10.688 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T13:08:11.689 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T13:08:11.689 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T13:08:11.761 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-09-16T13:08:11.784 DEBUG:teuthology.parallel:result is None 2024-09-16T13:08:11.784 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T13:08:11.797 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T13:08:11.797 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T13:08:11.799 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T13:08:11.806 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T13:08:11.812 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T13:08:11.822 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T13:08:11.823 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T13:08:11.825 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T13:08:11.868 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T13:08:11.874 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T13:08:11.875 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T13:08:11.877 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T13:08:11.895 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T13:08:11.901 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T13:08:11.904 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T13:08:11.912 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T13:08:11.914 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T13:08:11.921 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T13:08:11.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T13:08:11.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T13:08:11.940 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T13:08:11.971 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T13:08:11.971 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T13:08:11.981 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T13:08:11.988 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T13:08:11.994 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T13:08:12.000 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:08:12.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T13:08:12.065 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T13:08:12.073 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T13:08:12.080 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:08:12.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T13:08:12.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-09-16T13:08:12.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-09-16T13:08:12.155 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T13:08:12.163 INFO:teuthology.task.internal:Starting timer... 2024-09-16T13:08:12.163 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T13:08:12.204 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T13:08:12.217 DEBUG:teuthology.task.selinux:Excluding smithi098: OS 'ubuntu' does not support SELinux 2024-09-16T13:08:12.217 DEBUG:teuthology.task.selinux:Excluding smithi151: OS 'ubuntu' does not support SELinux 2024-09-16T13:08:12.217 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T13:08:12.217 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T13:08:12.218 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T13:08:12.218 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T13:08:12.226 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T13:08:12.238 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T13:08:12.239 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi151.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T13:13:45.685 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi151.front.sepia.ceph.com')] 2024-09-16T13:13:45.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T13:13:45.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:13:45.786 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-16T13:13:45.906 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T13:13:45.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-16T13:13:45.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T13:13:46.004 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-09-16T13:13:46.126 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-16T13:13:46.126 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T13:13:46.137 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T13:13:46.137 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T13:13:46.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T13:13:46.140 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T13:13:46.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T13:13:46.174 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T13:13:46.175 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: Command line: ntpd -gq 2024-09-16T13:13:46.176 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: proto: precision = 0.255 usec (-22) 2024-09-16T13:13:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T13:13:46.176 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T13:13:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T13:13:46.178 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T13:13:46.178 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T13:13:46.178 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T13:13:46.178 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T13:13:46.178 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: Listen normally on 3 enp3s0f1 172.21.15.98:123 2024-09-16T13:13:46.178 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: Listen normally on 4 lo [::1]:123 2024-09-16T13:13:46.178 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d26b%5]:123 2024-09-16T13:13:46.178 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:46 ntpd[13700]: Listening on routing socket on fd #22 for interface updates 2024-09-16T13:13:46.202 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T13:13:46.202 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: Command line: ntpd -gq 2024-09-16T13:13:46.203 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: proto: precision = 0.141 usec (-23) 2024-09-16T13:13:46.204 INFO:teuthology.orchestra.run.smithi151.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T13:13:46.204 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T13:13:46.204 INFO:teuthology.orchestra.run.smithi151.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T13:13:46.205 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T13:13:46.205 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T13:13:46.205 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T13:13:46.206 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T13:13:46.206 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: Listen normally on 3 enp3s0f1 172.21.15.151:123 2024-09-16T13:13:46.206 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: Listen normally on 4 lo [::1]:123 2024-09-16T13:13:46.206 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f63%5]:123 2024-09-16T13:13:46.206 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:46 ntpd[13727]: Listening on routing socket on fd #22 for interface updates 2024-09-16T13:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:47 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:47 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:47 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:47 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:47 ntpd[13700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:47.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:47 ntpd[13700]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:47.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:47 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:47.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:47 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:47.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:47 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:47.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:47 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:47.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:47 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:47.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:47 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:48.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:48 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:48.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:48 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:48.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:48 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:48.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:48 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:48.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:48 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:48.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:48 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:49.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:49 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:49.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:49 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:49.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:49 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:49.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:49 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:49.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:49 ntpd[13700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:49.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:49 ntpd[13700]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:49.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:49 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:50 ntpd[13700]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:50.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:50 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:50.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:50 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:50.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:50 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:50.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:50 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:50.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:50 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:50.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:50 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:50.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:50 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:50.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:50 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:51.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:51 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:51.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:51 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:51.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:51 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:51.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:51 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:51.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:51 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:51.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:51 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:51.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:51 ntpd[13700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:51.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:51 ntpd[13700]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:51.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:51 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:52.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:52.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:52.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:52.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:52.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:52 ntpd[13700]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:52.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:52 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:52.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:52 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:52.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:52 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:52.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:52 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:52.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:52 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:52.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:52 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:53 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:53 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:53 ntpd[13700]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:53 ntpd[13700]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:53 ntpd[13700]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:53 ntpd[13700]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:53.177 INFO:teuthology.orchestra.run.smithi098.stdout:16 Sep 13:13:53 ntpd[13700]: ntpd: time slew +0.000813 s 2024-09-16T13:13:53.177 INFO:teuthology.orchestra.run.smithi098.stdout:ntpd: time slew +0.000813s 2024-09-16T13:13:53.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:53 ntpd[13700]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T13:13:53.177 INFO:teuthology.orchestra.run.smithi098.stderr:16 Sep 13:13:53 ntpd[13700]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T13:13:53.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:53 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:53.203 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:53 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:53.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:53 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:53.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:53 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:53.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:53 ntpd[13727]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T13:13:53.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:53 ntpd[13727]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T13:13:53.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:53 ntpd[13727]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T13:13:53.204 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:53 ntpd[13727]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T13:13:53.204 INFO:teuthology.orchestra.run.smithi151.stdout:16 Sep 13:13:53 ntpd[13727]: ntpd: time slew +0.001794 s 2024-09-16T13:13:53.204 INFO:teuthology.orchestra.run.smithi151.stdout:ntpd: time slew +0.001794s 2024-09-16T13:13:53.205 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:53 ntpd[13727]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T13:13:53.205 INFO:teuthology.orchestra.run.smithi151.stderr:16 Sep 13:13:53 ntpd[13727]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T13:13:53.249 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T13:13:53.249 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2024-09-16T13:13:53.249 INFO:teuthology.orchestra.run.smithi098.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T13:13:53.249 INFO:teuthology.orchestra.run.smithi098.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T13:13:53.249 INFO:teuthology.orchestra.run.smithi098.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T13:13:53.249 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T13:13:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T13:13:53.276 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================== 2024-09-16T13:13:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T13:13:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T13:13:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T13:13:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T13:13:53.277 INFO:teuthology.run_tasks:Running task install... 2024-09-16T13:13:53.286 DEBUG:teuthology.task.install:project ceph 2024-09-16T13:13:53.286 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-16T13:13:53.286 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-16T13:13:53.286 INFO:teuthology.task.install:Using flavor: default 2024-09-16T13:13:53.292 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T13:13:53.292 INFO:teuthology.task.install:extra packages: [] 2024-09-16T13:13:53.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-key list | grep Ceph 2024-09-16T13:13:53.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-key list | grep Ceph 2024-09-16T13:13:53.393 INFO:teuthology.orchestra.run.smithi151.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T13:13:53.393 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T13:13:53.453 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T13:13:53.453 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T13:13:53.454 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T13:13:53.455 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T13:13:53.460 INFO:teuthology.orchestra.run.smithi151.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T13:13:53.460 INFO:teuthology.orchestra.run.smithi151.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T13:13:53.460 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T13:13:53.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T13:13:53.603 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-16T13:13:53.603 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-16T13:13:53.607 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-16T13:13:53.607 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-16T13:13:53.713 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:13:53.713 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T13:13:53.720 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:13:53.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T13:13:53.730 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-09-16T13:13:53.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo apt-get update 2024-09-16T13:13:53.868 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T13:13:53.874 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T13:13:53.891 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T13:13:53.920 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-16T13:13:53.924 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T13:13:53.924 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T13:13:53.929 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T13:13:53.941 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T13:13:53.947 INFO:teuthology.orchestra.run.smithi151.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T13:13:53.965 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T13:13:53.978 INFO:teuthology.orchestra.run.smithi151.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-16T13:13:53.986 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-16T13:13:54.000 INFO:teuthology.orchestra.run.smithi151.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T13:13:54.011 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:8 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T13:13:54.019 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-16T13:13:54.024 INFO:teuthology.orchestra.run.smithi151.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T13:13:54.046 INFO:teuthology.orchestra.run.smithi151.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-16T13:13:54.077 INFO:teuthology.orchestra.run.smithi151.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-16T13:13:54.526 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 28.5 kB in 1s (44.9 kB/s) 2024-09-16T13:13:54.587 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 28.5 kB in 1s (44.9 kB/s) 2024-09-16T13:13:55.679 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-16T13:13:55.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal 2024-09-16T13:13:55.755 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2024-09-16T13:13:55.779 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-16T13:13:55.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal 2024-09-16T13:13:55.850 INFO:teuthology.orchestra.run.smithi151.stdout:Reading package lists... 2024-09-16T13:13:55.972 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-09-16T13:13:55.973 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-09-16T13:13:56.068 INFO:teuthology.orchestra.run.smithi151.stdout:Building dependency tree... 2024-09-16T13:13:56.068 INFO:teuthology.orchestra.run.smithi151.stdout:Reading state information... 2024-09-16T13:13:56.161 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T13:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T13:13:56.162 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T13:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-09-16T13:13:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T13:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T13:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T13:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T13:13:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-16T13:13:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T13:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-16T13:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-16T13:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-16T13:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: socat xmlstarlet 2024-09-16T13:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-09-16T13:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T13:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T13:13:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T13:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T13:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T13:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T13:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T13:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T13:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T13:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout:Recommended packages: 2024-09-16T13:13:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: btrfs-tools 2024-09-16T13:13:56.267 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T13:13:56.268 INFO:teuthology.orchestra.run.smithi151.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T13:13:56.268 INFO:teuthology.orchestra.run.smithi151.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T13:13:56.268 INFO:teuthology.orchestra.run.smithi151.stdout:The following additional packages will be installed: 2024-09-16T13:13:56.269 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T13:13:56.269 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T13:13:56.269 INFO:teuthology.orchestra.run.smithi151.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T13:13:56.269 INFO:teuthology.orchestra.run.smithi151.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T13:13:56.269 INFO:teuthology.orchestra.run.smithi151.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T13:13:56.269 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T13:13:56.270 INFO:teuthology.orchestra.run.smithi151.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-16T13:13:56.271 INFO:teuthology.orchestra.run.smithi151.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-16T13:13:56.271 INFO:teuthology.orchestra.run.smithi151.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-16T13:13:56.271 INFO:teuthology.orchestra.run.smithi151.stdout: socat xmlstarlet 2024-09-16T13:13:56.271 INFO:teuthology.orchestra.run.smithi151.stdout:Suggested packages: 2024-09-16T13:13:56.271 INFO:teuthology.orchestra.run.smithi151.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T13:13:56.271 INFO:teuthology.orchestra.run.smithi151.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T13:13:56.271 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T13:13:56.271 INFO:teuthology.orchestra.run.smithi151.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T13:13:56.271 INFO:teuthology.orchestra.run.smithi151.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T13:13:56.272 INFO:teuthology.orchestra.run.smithi151.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T13:13:56.272 INFO:teuthology.orchestra.run.smithi151.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T13:13:56.272 INFO:teuthology.orchestra.run.smithi151.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T13:13:56.272 INFO:teuthology.orchestra.run.smithi151.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T13:13:56.272 INFO:teuthology.orchestra.run.smithi151.stdout:Recommended packages: 2024-09-16T13:13:56.272 INFO:teuthology.orchestra.run.smithi151.stdout: btrfs-tools 2024-09-16T13:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-09-16T13:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T13:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T13:13:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T13:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T13:13:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T13:13:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T13:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-16T13:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T13:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T13:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T13:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T13:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-16T13:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages will be upgraded: 2024-09-16T13:13:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 librbd1 2024-09-16T13:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T13:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 201 MB of archives. 2024-09-16T13:13:56.353 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-16T13:13:56.354 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T13:13:56.394 INFO:teuthology.orchestra.run.smithi151.stdout:The following NEW packages will be installed: 2024-09-16T13:13:56.394 INFO:teuthology.orchestra.run.smithi151.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T13:13:56.394 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T13:13:56.394 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T13:13:56.394 INFO:teuthology.orchestra.run.smithi151.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T13:13:56.395 INFO:teuthology.orchestra.run.smithi151.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T13:13:56.395 INFO:teuthology.orchestra.run.smithi151.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T13:13:56.395 INFO:teuthology.orchestra.run.smithi151.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T13:13:56.395 INFO:teuthology.orchestra.run.smithi151.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-16T13:13:56.395 INFO:teuthology.orchestra.run.smithi151.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T13:13:56.396 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T13:13:56.397 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T13:13:56.397 INFO:teuthology.orchestra.run.smithi151.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-16T13:13:56.397 INFO:teuthology.orchestra.run.smithi151.stdout:The following packages will be upgraded: 2024-09-16T13:13:56.397 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 librbd1 2024-09-16T13:13:56.437 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T13:13:56.439 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T13:13:56.440 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-16T13:13:56.448 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T13:13:56.454 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T13:13:56.463 INFO:teuthology.orchestra.run.smithi151.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T13:13:56.463 INFO:teuthology.orchestra.run.smithi151.stdout:Need to get 201 MB of archives. 2024-09-16T13:13:56.463 INFO:teuthology.orchestra.run.smithi151.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-16T13:13:56.463 INFO:teuthology.orchestra.run.smithi151.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T13:13:56.478 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T13:13:56.479 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T13:13:56.486 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T13:13:56.516 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T13:13:56.519 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T13:13:56.525 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T13:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T13:13:56.527 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T13:13:56.529 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T13:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T13:13:56.530 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T13:13:56.531 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T13:13:56.534 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T13:13:56.546 INFO:teuthology.orchestra.run.smithi151.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-16T13:13:56.548 INFO:teuthology.orchestra.run.smithi151.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T13:13:56.549 INFO:teuthology.orchestra.run.smithi151.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T13:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T13:13:56.550 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T13:13:56.551 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T13:13:56.552 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T13:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T13:13:56.553 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T13:13:56.554 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T13:13:56.555 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T13:13:56.556 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T13:13:56.559 INFO:teuthology.orchestra.run.smithi151.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T13:13:56.565 INFO:teuthology.orchestra.run.smithi151.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T13:13:56.569 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T13:13:56.570 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T13:13:56.586 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T13:13:56.590 INFO:teuthology.orchestra.run.smithi151.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T13:13:56.591 INFO:teuthology.orchestra.run.smithi151.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T13:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T13:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T13:13:56.592 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T13:13:56.593 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T13:13:56.594 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T13:13:56.596 INFO:teuthology.orchestra.run.smithi151.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T13:13:56.596 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T13:13:56.597 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T13:13:56.604 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T13:13:56.605 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T13:13:56.609 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-16T13:13:56.622 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T13:13:56.623 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T13:13:56.623 INFO:teuthology.orchestra.run.smithi098.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T13:13:56.624 INFO:teuthology.orchestra.run.smithi098.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T13:13:56.630 INFO:teuthology.orchestra.run.smithi151.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T13:13:56.633 INFO:teuthology.orchestra.run.smithi151.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T13:13:56.640 INFO:teuthology.orchestra.run.smithi151.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T13:13:56.642 INFO:teuthology.orchestra.run.smithi151.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T13:13:56.643 INFO:teuthology.orchestra.run.smithi151.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T13:13:56.644 INFO:teuthology.orchestra.run.smithi151.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T13:13:56.645 INFO:teuthology.orchestra.run.smithi151.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T13:13:56.645 INFO:teuthology.orchestra.run.smithi151.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T13:13:56.645 INFO:teuthology.orchestra.run.smithi151.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T13:13:56.653 INFO:teuthology.orchestra.run.smithi151.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T13:13:56.662 INFO:teuthology.orchestra.run.smithi098.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-16T13:13:56.669 INFO:teuthology.orchestra.run.smithi151.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T13:13:56.669 INFO:teuthology.orchestra.run.smithi151.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T13:13:56.670 INFO:teuthology.orchestra.run.smithi151.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T13:13:56.670 INFO:teuthology.orchestra.run.smithi151.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T13:13:56.672 INFO:teuthology.orchestra.run.smithi151.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T13:13:56.672 INFO:teuthology.orchestra.run.smithi151.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T13:13:56.673 INFO:teuthology.orchestra.run.smithi151.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T13:13:56.675 INFO:teuthology.orchestra.run.smithi151.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T13:13:56.675 INFO:teuthology.orchestra.run.smithi098.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-16T13:13:56.675 INFO:teuthology.orchestra.run.smithi151.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T13:13:56.679 INFO:teuthology.orchestra.run.smithi098.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-16T13:13:56.680 INFO:teuthology.orchestra.run.smithi098.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-16T13:13:56.682 INFO:teuthology.orchestra.run.smithi098.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-16T13:13:56.683 INFO:teuthology.orchestra.run.smithi098.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-16T13:13:56.690 INFO:teuthology.orchestra.run.smithi098.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-16T13:13:56.692 INFO:teuthology.orchestra.run.smithi151.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T13:13:56.694 INFO:teuthology.orchestra.run.smithi151.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T13:13:56.695 INFO:teuthology.orchestra.run.smithi151.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T13:13:56.701 INFO:teuthology.orchestra.run.smithi151.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T13:13:56.701 INFO:teuthology.orchestra.run.smithi151.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T13:13:56.702 INFO:teuthology.orchestra.run.smithi151.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T13:13:56.712 INFO:teuthology.orchestra.run.smithi151.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T13:13:56.712 INFO:teuthology.orchestra.run.smithi151.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T13:13:56.713 INFO:teuthology.orchestra.run.smithi098.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T13:13:56.715 INFO:teuthology.orchestra.run.smithi151.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T13:13:56.716 INFO:teuthology.orchestra.run.smithi151.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T13:13:56.729 INFO:teuthology.orchestra.run.smithi098.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T13:13:56.730 INFO:teuthology.orchestra.run.smithi098.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T13:13:56.731 INFO:teuthology.orchestra.run.smithi151.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T13:13:56.733 INFO:teuthology.orchestra.run.smithi151.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T13:13:56.734 INFO:teuthology.orchestra.run.smithi151.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T13:13:56.736 INFO:teuthology.orchestra.run.smithi151.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T13:13:56.736 INFO:teuthology.orchestra.run.smithi151.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T13:13:56.737 INFO:teuthology.orchestra.run.smithi151.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T13:13:56.745 INFO:teuthology.orchestra.run.smithi098.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T13:13:56.745 INFO:teuthology.orchestra.run.smithi098.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T13:13:56.813 INFO:teuthology.orchestra.run.smithi151.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T13:13:56.823 INFO:teuthology.orchestra.run.smithi098.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-16T13:13:56.823 INFO:teuthology.orchestra.run.smithi098.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T13:13:56.824 INFO:teuthology.orchestra.run.smithi098.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T13:13:56.824 INFO:teuthology.orchestra.run.smithi098.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-16T13:13:56.825 INFO:teuthology.orchestra.run.smithi098.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T13:13:56.826 INFO:teuthology.orchestra.run.smithi098.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T13:13:56.827 INFO:teuthology.orchestra.run.smithi098.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T13:13:56.827 INFO:teuthology.orchestra.run.smithi151.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T13:13:56.829 INFO:teuthology.orchestra.run.smithi151.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T13:13:56.829 INFO:teuthology.orchestra.run.smithi098.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T13:13:56.830 INFO:teuthology.orchestra.run.smithi098.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T13:13:56.830 INFO:teuthology.orchestra.run.smithi098.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-16T13:13:56.840 INFO:teuthology.orchestra.run.smithi151.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T13:13:56.842 INFO:teuthology.orchestra.run.smithi098.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T13:13:56.844 INFO:teuthology.orchestra.run.smithi098.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T13:13:56.846 INFO:teuthology.orchestra.run.smithi098.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T13:13:56.861 INFO:teuthology.orchestra.run.smithi098.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T13:13:56.862 INFO:teuthology.orchestra.run.smithi098.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T13:13:56.865 INFO:teuthology.orchestra.run.smithi098.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T13:13:56.869 INFO:teuthology.orchestra.run.smithi098.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T13:13:56.884 INFO:teuthology.orchestra.run.smithi098.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T13:13:56.887 INFO:teuthology.orchestra.run.smithi098.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T13:13:56.902 INFO:teuthology.orchestra.run.smithi151.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T13:13:56.903 INFO:teuthology.orchestra.run.smithi151.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T13:13:56.904 INFO:teuthology.orchestra.run.smithi151.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T13:13:56.904 INFO:teuthology.orchestra.run.smithi151.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T13:13:56.905 INFO:teuthology.orchestra.run.smithi151.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T13:13:56.906 INFO:teuthology.orchestra.run.smithi151.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T13:13:56.907 INFO:teuthology.orchestra.run.smithi151.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T13:13:56.908 INFO:teuthology.orchestra.run.smithi151.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T13:13:56.924 INFO:teuthology.orchestra.run.smithi151.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T13:13:56.929 INFO:teuthology.orchestra.run.smithi098.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T13:13:56.929 INFO:teuthology.orchestra.run.smithi151.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T13:13:56.930 INFO:teuthology.orchestra.run.smithi098.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T13:13:56.930 INFO:teuthology.orchestra.run.smithi098.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T13:13:56.930 INFO:teuthology.orchestra.run.smithi098.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T13:13:56.931 INFO:teuthology.orchestra.run.smithi098.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T13:13:56.931 INFO:teuthology.orchestra.run.smithi151.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T13:13:56.932 INFO:teuthology.orchestra.run.smithi098.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T13:13:56.934 INFO:teuthology.orchestra.run.smithi151.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T13:13:56.935 INFO:teuthology.orchestra.run.smithi151.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T13:13:56.940 INFO:teuthology.orchestra.run.smithi151.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T13:13:56.942 INFO:teuthology.orchestra.run.smithi151.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T13:13:56.943 INFO:teuthology.orchestra.run.smithi098.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T13:13:56.944 INFO:teuthology.orchestra.run.smithi098.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T13:13:56.944 INFO:teuthology.orchestra.run.smithi098.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T13:13:56.952 INFO:teuthology.orchestra.run.smithi098.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T13:13:56.960 INFO:teuthology.orchestra.run.smithi151.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T13:13:56.962 INFO:teuthology.orchestra.run.smithi151.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T13:13:56.972 INFO:teuthology.orchestra.run.smithi098.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T13:13:56.972 INFO:teuthology.orchestra.run.smithi098.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T13:13:56.973 INFO:teuthology.orchestra.run.smithi098.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T13:13:56.975 INFO:teuthology.orchestra.run.smithi098.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T13:13:56.976 INFO:teuthology.orchestra.run.smithi098.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T13:13:57.004 INFO:teuthology.orchestra.run.smithi151.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T13:13:57.005 INFO:teuthology.orchestra.run.smithi151.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T13:13:57.005 INFO:teuthology.orchestra.run.smithi151.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T13:13:57.005 INFO:teuthology.orchestra.run.smithi151.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T13:13:57.006 INFO:teuthology.orchestra.run.smithi151.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T13:13:57.007 INFO:teuthology.orchestra.run.smithi151.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T13:13:57.018 INFO:teuthology.orchestra.run.smithi151.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T13:13:57.018 INFO:teuthology.orchestra.run.smithi151.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T13:13:57.018 INFO:teuthology.orchestra.run.smithi151.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T13:13:57.021 INFO:teuthology.orchestra.run.smithi151.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T13:13:57.038 INFO:teuthology.orchestra.run.smithi151.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T13:13:57.039 INFO:teuthology.orchestra.run.smithi151.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T13:13:57.040 INFO:teuthology.orchestra.run.smithi151.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T13:13:57.042 INFO:teuthology.orchestra.run.smithi151.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T13:13:57.043 INFO:teuthology.orchestra.run.smithi151.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T13:13:57.349 INFO:teuthology.orchestra.run.smithi098.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-16T13:13:57.393 INFO:teuthology.orchestra.run.smithi098.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-16T13:13:57.395 INFO:teuthology.orchestra.run.smithi098.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-16T13:13:57.396 INFO:teuthology.orchestra.run.smithi098.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-16T13:13:57.422 INFO:teuthology.orchestra.run.smithi098.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-16T13:13:57.503 INFO:teuthology.orchestra.run.smithi098.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-16T13:13:57.521 INFO:teuthology.orchestra.run.smithi151.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-16T13:13:58.011 INFO:teuthology.orchestra.run.smithi098.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-16T13:13:58.012 INFO:teuthology.orchestra.run.smithi098.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-16T13:13:58.014 INFO:teuthology.orchestra.run.smithi151.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-16T13:13:58.025 INFO:teuthology.orchestra.run.smithi098.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-16T13:13:58.039 INFO:teuthology.orchestra.run.smithi151.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-16T13:13:58.042 INFO:teuthology.orchestra.run.smithi098.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-16T13:13:58.043 INFO:teuthology.orchestra.run.smithi098.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-16T13:13:58.044 INFO:teuthology.orchestra.run.smithi098.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-16T13:13:58.067 INFO:teuthology.orchestra.run.smithi151.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-16T13:13:58.069 INFO:teuthology.orchestra.run.smithi151.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-16T13:13:58.094 INFO:teuthology.orchestra.run.smithi151.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-16T13:13:58.105 INFO:teuthology.orchestra.run.smithi151.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-16T13:13:58.148 INFO:teuthology.orchestra.run.smithi098.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-16T13:13:58.152 INFO:teuthology.orchestra.run.smithi151.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-16T13:13:58.300 INFO:teuthology.orchestra.run.smithi098.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-16T13:13:58.300 INFO:teuthology.orchestra.run.smithi098.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-16T13:13:58.673 INFO:teuthology.orchestra.run.smithi151.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-16T13:13:58.675 INFO:teuthology.orchestra.run.smithi151.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-16T13:13:58.729 INFO:teuthology.orchestra.run.smithi151.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-16T13:13:59.692 INFO:teuthology.orchestra.run.smithi098.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-16T13:13:59.744 INFO:teuthology.orchestra.run.smithi098.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-16T13:13:59.745 INFO:teuthology.orchestra.run.smithi098.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-16T13:13:59.873 INFO:teuthology.orchestra.run.smithi151.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-16T13:14:00.033 INFO:teuthology.orchestra.run.smithi151.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-16T13:14:00.035 INFO:teuthology.orchestra.run.smithi151.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-16T13:14:00.036 INFO:teuthology.orchestra.run.smithi151.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-16T13:14:00.066 INFO:teuthology.orchestra.run.smithi151.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-16T13:14:00.181 INFO:teuthology.orchestra.run.smithi151.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-16T13:14:00.227 INFO:teuthology.orchestra.run.smithi098.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-16T13:14:00.638 INFO:teuthology.orchestra.run.smithi151.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-16T13:14:00.639 INFO:teuthology.orchestra.run.smithi151.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-16T13:14:00.651 INFO:teuthology.orchestra.run.smithi151.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-16T13:14:00.668 INFO:teuthology.orchestra.run.smithi151.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-16T13:14:00.669 INFO:teuthology.orchestra.run.smithi151.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-16T13:14:00.669 INFO:teuthology.orchestra.run.smithi151.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-16T13:14:00.763 INFO:teuthology.orchestra.run.smithi151.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-16T13:14:00.877 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 201 MB in 4s (51.2 MB/s) 2024-09-16T13:14:00.892 INFO:teuthology.orchestra.run.smithi151.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-16T13:14:00.892 INFO:teuthology.orchestra.run.smithi151.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-16T13:14:01.097 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T13:14:01.153 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T13:14:01.157 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T13:14:01.254 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T13:14:01.607 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:01.733 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T13:14:01.986 INFO:teuthology.orchestra.run.smithi151.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-16T13:14:01.990 INFO:teuthology.orchestra.run.smithi151.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-16T13:14:01.991 INFO:teuthology.orchestra.run.smithi151.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-16T13:14:02.322 INFO:teuthology.orchestra.run.smithi151.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-16T13:14:02.413 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:02.537 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T13:14:02.980 INFO:teuthology.orchestra.run.smithi151.stdout:Fetched 201 MB in 6s (34.0 MB/s) 2024-09-16T13:14:03.133 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T13:14:03.189 INFO:teuthology.orchestra.run.smithi151.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T13:14:03.193 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T13:14:03.216 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T13:14:03.236 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:03.265 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T13:14:03.266 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:03.560 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rados. 2024-09-16T13:14:03.576 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:03.618 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:03.628 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:03.753 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T13:14:03.988 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T13:14:04.008 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:04.046 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:04.299 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T13:14:04.317 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:04.357 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:04.466 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:04.592 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T13:14:04.685 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T13:14:04.702 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:04.734 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:05.046 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T13:14:05.062 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T13:14:05.104 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T13:14:05.280 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T13:14:05.296 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:05.347 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:05.357 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T13:14:05.374 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:05.415 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:05.700 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rados. 2024-09-16T13:14:05.716 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:05.758 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:05.810 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T13:14:05.828 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T13:14:05.868 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T13:14:06.086 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T13:14:06.106 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:06.145 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:06.163 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T13:14:06.178 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T13:14:06.213 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T13:14:06.398 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T13:14:06.415 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:06.456 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:06.582 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T13:14:06.599 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T13:14:06.641 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T13:14:06.784 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T13:14:06.801 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:06.851 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:07.010 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T13:14:07.028 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T13:14:07.068 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T13:14:07.138 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T13:14:07.157 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T13:14:07.196 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T13:14:07.355 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T13:14:07.373 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T13:14:07.405 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T13:14:07.450 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T13:14:07.465 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:07.508 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:07.858 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T13:14:07.870 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T13:14:07.874 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T13:14:07.887 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T13:14:07.927 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:14:07.928 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T13:14:08.249 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T13:14:08.266 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T13:14:08.307 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T13:14:08.453 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T13:14:08.470 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T13:14:08.511 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:14:08.652 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T13:14:08.673 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T13:14:08.710 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T13:14:08.866 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T13:14:08.882 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T13:14:08.925 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:14:09.039 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T13:14:09.055 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T13:14:09.088 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T13:14:09.327 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T13:14:09.347 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T13:14:09.386 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T13:14:09.451 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T13:14:09.467 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T13:14:09.509 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T13:14:09.755 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librgw2. 2024-09-16T13:14:09.771 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:09.813 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:09.946 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T13:14:09.966 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T13:14:10.015 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:14:10.559 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T13:14:10.566 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T13:14:10.576 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T13:14:10.582 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:10.617 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:14:10.624 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:10.969 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liboath0. 2024-09-16T13:14:10.979 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T13:14:10.986 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T13:14:10.996 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T13:14:11.027 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T13:14:11.037 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:14:11.338 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T13:14:11.355 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:11.397 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:11.425 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T13:14:11.443 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T13:14:11.483 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T13:14:11.836 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package librgw2. 2024-09-16T13:14:11.855 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:11.859 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-common. 2024-09-16T13:14:11.874 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:11.894 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:11.916 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:12.648 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T13:14:12.666 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:12.706 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:13.059 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liboath0. 2024-09-16T13:14:13.077 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T13:14:13.118 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T13:14:13.404 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T13:14:13.423 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:13.454 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:13.950 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-common. 2024-09-16T13:14:13.967 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:14.008 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:14.489 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-base. 2024-09-16T13:14:14.506 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:14.558 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:15.394 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T13:14:15.412 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T13:14:15.444 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T13:14:15.839 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T13:14:15.857 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T13:14:15.931 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T13:14:16.259 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T13:14:16.274 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-16T13:14:16.317 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T13:14:16.570 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T13:14:16.582 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-base. 2024-09-16T13:14:16.586 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:16.597 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:16.628 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:16.642 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:17.033 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T13:14:17.052 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T13:14:17.091 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T13:14:17.394 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T13:14:17.410 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T13:14:17.452 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T13:14:17.504 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T13:14:17.522 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T13:14:17.554 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T13:14:17.797 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T13:14:17.817 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T13:14:17.855 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T13:14:17.877 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T13:14:17.894 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T13:14:17.926 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T13:14:18.167 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T13:14:18.185 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T13:14:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T13:14:18.254 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T13:14:18.270 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-16T13:14:18.312 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T13:14:18.520 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T13:14:18.536 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T13:14:18.548 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T13:14:18.565 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:18.570 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T13:14:18.606 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:18.873 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T13:14:18.890 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T13:14:18.940 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T13:14:19.019 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T13:14:19.038 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T13:14:19.077 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T13:14:19.243 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-mako. 2024-09-16T13:14:19.259 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T13:14:19.300 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T13:14:19.380 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T13:14:19.397 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T13:14:19.438 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T13:14:19.662 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T13:14:19.681 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T13:14:19.720 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T13:14:19.758 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T13:14:19.777 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T13:14:19.816 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T13:14:20.023 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T13:14:20.037 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T13:14:20.090 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T13:14:20.153 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T13:14:20.171 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T13:14:20.211 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T13:14:20.393 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webob. 2024-09-16T13:14:20.410 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T13:14:20.451 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T13:14:20.522 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T13:14:20.539 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T13:14:20.580 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T13:14:20.787 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T13:14:20.805 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T13:14:20.837 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T13:14:20.908 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T13:14:20.924 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T13:14:20.966 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T13:14:21.138 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T13:14:21.154 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T13:14:21.196 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T13:14:21.269 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-mako. 2024-09-16T13:14:21.289 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T13:14:21.327 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T13:14:21.558 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T13:14:21.574 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T13:14:21.620 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T13:14:21.655 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T13:14:21.674 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T13:14:21.713 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T13:14:21.936 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T13:14:21.954 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T13:14:21.995 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T13:14:22.016 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T13:14:22.032 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T13:14:22.074 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T13:14:22.281 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-paste. 2024-09-16T13:14:22.300 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T13:14:22.339 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T13:14:22.360 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-webob. 2024-09-16T13:14:22.376 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T13:14:22.443 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T13:14:22.642 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T13:14:22.661 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T13:14:22.700 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T13:14:22.754 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T13:14:22.774 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T13:14:22.813 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T13:14:23.003 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T13:14:23.019 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T13:14:23.061 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T13:14:23.115 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T13:14:23.131 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T13:14:23.173 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T13:14:23.373 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T13:14:23.392 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T13:14:23.431 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T13:14:23.543 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T13:14:23.562 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T13:14:23.604 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T13:14:23.751 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T13:14:23.769 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T13:14:23.809 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T13:14:23.938 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T13:14:23.955 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T13:14:23.996 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T13:14:24.160 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T13:14:24.177 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T13:14:24.237 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T13:14:24.323 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-paste. 2024-09-16T13:14:24.339 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T13:14:24.382 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T13:14:24.571 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T13:14:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:24.629 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:24.723 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T13:14:24.739 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T13:14:24.781 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T13:14:24.990 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T13:14:25.006 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:25.048 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:25.117 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T13:14:25.137 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T13:14:25.175 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T13:14:25.486 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T13:14:25.503 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T13:14:25.535 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T13:14:25.544 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T13:14:25.551 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:25.593 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:25.856 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T13:14:25.872 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T13:14:25.922 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T13:14:26.277 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T13:14:26.293 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T13:14:26.335 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T13:14:26.372 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T13:14:26.390 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:26.430 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:26.672 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T13:14:26.688 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:26.755 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:27.108 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T13:14:27.124 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:27.166 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:27.698 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T13:14:27.713 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:27.756 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:28.609 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T13:14:28.629 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:28.667 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:29.217 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph. 2024-09-16T13:14:29.237 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:29.275 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:29.578 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T13:14:29.596 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:29.636 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:30.081 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T13:14:30.097 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:30.164 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:30.701 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephadm. 2024-09-16T13:14:30.720 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:30.784 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:30.944 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph. 2024-09-16T13:14:30.960 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:31.002 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:31.103 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T13:14:31.121 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T13:14:31.161 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T13:14:31.354 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T13:14:31.371 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:31.412 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:31.481 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T13:14:31.498 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:31.539 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:31.874 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T13:14:31.889 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:31.901 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T13:14:31.920 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T13:14:31.932 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:31.959 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T13:14:32.178 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T13:14:32.197 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T13:14:32.228 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T13:14:32.468 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package cephadm. 2024-09-16T13:14:32.486 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:32.488 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T13:14:32.506 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T13:14:32.535 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:32.547 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T13:14:32.846 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T13:14:32.862 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T13:14:32.899 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-routes. 2024-09-16T13:14:32.904 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T13:14:32.917 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-16T13:14:32.957 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T13:14:33.261 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T13:14:33.274 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T13:14:33.280 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:33.291 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:33.319 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:33.332 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:33.694 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T13:14:33.713 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T13:14:33.751 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T13:14:34.012 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T13:14:34.028 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T13:14:34.087 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T13:14:34.356 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T13:14:34.372 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T13:14:34.414 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T13:14:34.708 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T13:14:34.728 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T13:14:34.734 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-routes. 2024-09-16T13:14:34.753 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-16T13:14:34.766 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T13:14:34.792 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T13:14:35.070 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T13:14:35.086 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T13:14:35.128 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T13:14:35.129 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T13:14:35.144 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:35.186 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:35.473 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T13:14:35.490 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T13:14:35.533 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T13:14:36.633 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T13:14:36.653 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T13:14:36.691 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T13:14:36.840 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T13:14:36.856 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T13:14:36.898 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T13:14:36.994 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T13:14:37.010 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T13:14:37.052 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T13:14:37.413 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T13:14:37.429 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T13:14:37.452 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T13:14:37.468 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T13:14:37.471 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T13:14:37.510 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T13:14:37.855 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T13:14:37.874 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T13:14:37.913 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T13:14:38.630 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T13:14:38.647 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:38.688 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:38.856 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T13:14:38.873 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T13:14:38.914 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T13:14:39.476 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T13:14:39.492 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T13:14:39.533 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T13:14:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T13:14:39.828 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T13:14:39.866 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T13:14:39.937 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T13:14:39.955 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T13:14:39.995 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T13:14:40.194 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T13:14:40.213 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T13:14:40.252 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T13:14:40.580 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T13:14:40.596 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T13:14:40.638 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T13:14:40.703 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T13:14:40.720 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:40.794 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:40.933 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T13:14:40.952 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-16T13:14:40.990 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T13:14:41.353 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T13:14:41.369 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T13:14:41.410 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T13:14:41.731 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T13:14:41.750 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T13:14:41.789 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T13:14:41.922 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T13:14:41.940 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T13:14:41.980 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T13:14:42.109 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T13:14:42.126 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T13:14:42.167 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T13:14:42.266 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T13:14:42.286 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T13:14:42.324 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T13:14:42.504 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T13:14:42.522 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T13:14:42.562 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T13:14:42.677 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T13:14:42.698 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T13:14:42.736 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T13:14:42.857 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T13:14:42.873 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T13:14:42.932 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T13:14:43.046 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T13:14:43.066 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-16T13:14:43.105 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T13:14:43.269 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T13:14:43.286 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T13:14:43.327 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T13:14:43.441 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T13:14:43.459 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T13:14:43.499 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T13:14:43.811 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T13:14:43.826 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T13:14:43.868 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T13:14:43.946 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T13:14:43.965 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:44.004 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:44.188 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T13:14:44.204 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T13:14:44.262 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T13:14:44.324 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T13:14:44.340 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T13:14:44.382 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T13:14:44.583 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T13:14:44.600 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T13:14:44.640 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T13:14:44.961 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T13:14:44.977 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T13:14:45.019 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T13:14:45.102 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T13:14:45.118 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T13:14:45.160 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T13:14:45.389 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T13:14:45.407 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T13:14:45.446 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package jq. 2024-09-16T13:14:45.447 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T13:14:45.465 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T13:14:45.504 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T13:14:45.782 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package socat. 2024-09-16T13:14:45.799 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T13:14:45.840 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T13:14:46.139 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T13:14:46.158 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T13:14:46.197 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T13:14:46.355 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T13:14:46.371 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:46.413 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:46.531 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-test. 2024-09-16T13:14:46.549 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:46.589 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:46.707 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T13:14:46.726 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T13:14:46.765 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T13:14:47.135 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T13:14:47.151 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T13:14:47.193 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T13:14:47.474 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package jq. 2024-09-16T13:14:47.490 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T13:14:47.532 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T13:14:47.826 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package socat. 2024-09-16T13:14:47.843 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T13:14:47.884 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T13:14:48.188 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T13:14:48.207 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T13:14:48.246 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T13:14:48.617 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-test. 2024-09-16T13:14:48.638 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:48.675 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:51.252 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T13:14:51.271 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:51.310 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:51.651 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T13:14:51.669 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T13:14:51.709 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T13:14:52.189 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T13:14:52.204 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:52.246 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:52.666 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T13:14:52.683 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T13:14:52.742 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T13:14:53.080 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T13:14:53.096 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T13:14:53.138 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T13:14:53.263 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T13:14:53.280 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T13:14:53.313 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:53.640 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T13:14:53.657 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T13:14:53.698 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T13:14:54.227 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T13:14:54.244 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:14:54.285 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:14:54.706 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T13:14:54.726 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T13:14:54.783 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T13:14:54.785 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T13:14:54.807 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T13:14:54.852 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T13:14:55.128 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T13:14:55.147 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T13:14:55.186 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T13:14:55.221 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T13:14:55.239 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T13:14:55.279 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T13:14:55.591 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-attr. 2024-09-16T13:14:55.608 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-16T13:14:55.649 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T13:14:55.985 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T13:14:56.005 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T13:14:56.051 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T13:14:56.354 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T13:14:56.372 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T13:14:56.412 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T13:14:56.732 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T13:14:56.753 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T13:14:56.790 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T13:14:56.806 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T13:14:56.823 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T13:14:56.864 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T13:14:57.184 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T13:14:57.201 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T13:14:57.242 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T13:14:57.269 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T13:14:57.289 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T13:14:57.327 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T13:14:57.553 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-attr. 2024-09-16T13:14:57.569 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-16T13:14:57.611 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T13:14:57.647 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T13:14:57.664 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T13:14:57.705 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T13:14:57.930 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T13:14:57.948 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T13:14:57.988 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T13:14:58.042 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T13:14:58.059 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T13:14:58.100 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T13:14:58.316 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T13:14:58.333 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T13:14:58.374 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T13:14:58.428 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-py. 2024-09-16T13:14:58.445 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T13:14:58.486 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T13:14:58.702 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T13:14:58.723 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T13:14:58.760 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T13:14:58.814 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T13:14:58.832 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T13:14:58.906 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T13:14:59.217 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T13:14:59.222 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T13:14:59.235 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T13:14:59.240 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T13:14:59.275 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T13:14:59.280 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T13:14:59.545 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T13:14:59.566 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T13:14:59.603 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T13:14:59.609 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T13:14:59.625 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T13:14:59.667 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T13:14:59.965 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T13:14:59.983 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T13:14:59.995 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T13:15:00.011 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T13:15:00.031 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T13:15:00.053 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T13:15:00.293 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T13:15:00.311 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T13:15:00.351 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T13:15:00.389 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-py. 2024-09-16T13:15:00.408 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T13:15:00.447 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T13:15:00.793 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T13:15:00.810 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T13:15:00.842 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T13:15:01.100 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package radosgw. 2024-09-16T13:15:01.116 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:15:01.145 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T13:15:01.158 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:01.161 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T13:15:01.203 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T13:15:01.514 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T13:15:01.531 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T13:15:01.572 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T13:15:01.926 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T13:15:01.942 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T13:15:01.984 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T13:15:02.262 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T13:15:02.279 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T13:15:02.320 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T13:15:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T13:15:02.970 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:15:03.011 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:03.074 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package radosgw. 2024-09-16T13:15:03.092 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:15:03.140 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:03.312 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T13:15:03.600 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T13:15:03.751 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T13:15:03.877 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T13:15:04.071 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:04.725 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user cephadm....done 2024-09-16T13:15:04.870 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T13:15:04.918 INFO:teuthology.orchestra.run.smithi151.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T13:15:04.936 INFO:teuthology.orchestra.run.smithi151.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T13:15:04.971 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T13:15:04.976 INFO:teuthology.orchestra.run.smithi151.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:05.315 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T13:15:05.319 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T13:15:05.600 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T13:15:05.609 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T13:15:05.725 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T13:15:05.728 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T13:15:05.853 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T13:15:05.853 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T13:15:06.071 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:06.163 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T13:15:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T13:15:06.633 INFO:teuthology.orchestra.run.smithi151.stdout:Adding system user cephadm....done 2024-09-16T13:15:06.827 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T13:15:06.850 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T13:15:06.962 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T13:15:07.143 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T13:15:07.305 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T13:15:07.427 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T13:15:07.599 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T13:15:07.720 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:07.760 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T13:15:07.886 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T13:15:07.996 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T13:15:08.097 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T13:15:08.204 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T13:15:08.390 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T13:15:08.639 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T13:15:08.674 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T13:15:08.801 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T13:15:08.932 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T13:15:09.177 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T13:15:09.217 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T13:15:09.501 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T13:15:09.503 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T13:15:09.779 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T13:15:09.793 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:10.078 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T13:15:10.147 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T13:15:10.213 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T13:15:10.424 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T13:15:10.532 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T13:15:10.741 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T13:15:10.824 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T13:15:10.950 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T13:15:11.026 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T13:15:11.326 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T13:15:11.352 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T13:15:11.478 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T13:15:11.587 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T13:15:11.619 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T13:15:11.855 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T13:15:11.887 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T13:15:12.224 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T13:15:12.263 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T13:15:12.547 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T13:15:12.592 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T13:15:12.874 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T13:15:12.961 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T13:15:13.087 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:13.166 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T13:15:13.388 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:15:13.467 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T13:15:13.497 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T13:15:13.593 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T13:15:13.744 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T13:15:13.866 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T13:15:14.009 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T13:15:14.028 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T13:15:14.135 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T13:15:14.428 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T13:15:14.430 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T13:15:14.769 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:15:14.798 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T13:15:14.880 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T13:15:14.981 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T13:15:15.159 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T13:15:15.284 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:15.291 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T13:15:15.611 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:15:15.745 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T13:15:15.876 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T13:15:16.080 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T13:15:16.177 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T13:15:16.206 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T13:15:16.303 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T13:15:16.357 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T13:15:16.596 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T13:15:16.650 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T13:15:16.872 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T13:15:17.051 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:15:17.173 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T13:15:17.536 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T13:15:17.669 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T13:15:17.786 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T13:15:17.794 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T13:15:17.912 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T13:15:18.138 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T13:15:18.247 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T13:15:18.447 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T13:15:18.632 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T13:15:18.740 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T13:15:18.749 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T13:15:18.884 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T13:15:18.907 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T13:15:19.184 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T13:15:19.218 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T13:15:19.468 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T13:15:19.539 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:19.670 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T13:15:19.777 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T13:15:20.206 INFO:teuthology.orchestra.run.smithi151.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T13:15:20.447 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T13:15:20.573 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T13:15:20.932 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T13:15:21.135 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:21.261 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T13:15:21.300 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T13:15:21.412 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T13:15:21.426 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T13:15:21.585 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T13:15:21.780 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T13:15:21.928 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T13:15:22.073 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T13:15:22.221 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:22.357 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:15:22.372 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T13:15:22.483 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T13:15:22.835 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:22.961 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T13:15:23.296 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:23.422 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T13:15:23.757 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T13:15:23.836 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:23.962 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T13:15:24.113 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T13:15:24.481 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T13:15:24.787 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:24.790 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T13:15:24.913 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T13:15:25.039 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T13:15:25.075 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T13:15:25.192 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T13:15:25.192 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T13:15:25.308 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T13:15:25.519 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:25.618 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T13:15:25.645 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T13:15:25.744 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T13:15:25.972 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:26.070 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:26.097 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T13:15:26.196 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:26.423 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T13:15:26.462 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T13:15:26.462 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T13:15:26.882 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:27.234 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T13:15:27.394 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T13:15:27.481 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:27.632 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T13:15:27.712 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T13:15:27.758 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T13:15:27.884 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T13:15:28.010 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T13:15:28.073 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:28.191 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:28.311 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T13:15:28.317 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:28.443 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:28.470 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T13:15:28.586 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T13:15:28.771 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:28.888 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T13:15:28.897 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:29.121 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T13:15:29.121 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T13:15:29.172 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:29.298 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:29.408 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T13:15:29.565 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:29.769 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T13:15:30.070 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T13:15:30.336 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:30.421 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T13:15:30.790 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:30.891 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:30.897 INFO:teuthology.orchestra.run.smithi098.stdout:Adding group ceph....done 2024-09-16T13:15:31.026 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:31.152 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:31.258 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user ceph....done 2024-09-16T13:15:31.277 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T13:15:31.451 INFO:teuthology.orchestra.run.smithi098.stdout:Setting system user ceph properties....done 2024-09-16T13:15:31.472 INFO:teuthology.orchestra.run.smithi098.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T13:15:31.579 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T13:15:31.613 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T13:15:31.889 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:31.939 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T13:15:32.015 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:32.132 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T13:15:32.413 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:32.560 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:32.811 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T13:15:32.812 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T13:15:33.036 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:33.500 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:33.588 INFO:teuthology.orchestra.run.smithi151.stdout:Adding group ceph....done 2024-09-16T13:15:33.873 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T13:15:33.990 INFO:teuthology.orchestra.run.smithi151.stdout:Adding system user ceph....done 2024-09-16T13:15:34.167 INFO:teuthology.orchestra.run.smithi151.stdout:Setting system user ceph properties....done 2024-09-16T13:15:34.193 INFO:teuthology.orchestra.run.smithi151.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T13:15:34.327 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:34.348 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T13:15:34.571 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T13:15:34.571 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T13:15:34.671 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T13:15:35.029 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:35.129 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:35.255 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:35.256 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T13:15:35.257 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T13:15:35.546 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T13:15:35.546 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T13:15:35.705 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:36.199 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T13:15:36.209 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T13:15:36.595 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T13:15:36.649 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:36.775 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:36.918 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:37.067 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:37.146 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T13:15:37.146 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T13:15:37.302 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T13:15:37.302 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T13:15:37.619 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:37.752 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:37.762 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:37.888 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:37.986 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T13:15:37.986 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T13:15:38.031 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:38.192 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T13:15:38.343 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T13:15:38.420 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:38.628 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T13:15:38.913 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T13:15:38.913 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T13:15:39.347 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:39.448 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:39.591 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:39.842 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T13:15:39.842 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T13:15:40.284 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:40.427 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:40.577 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:40.720 INFO:teuthology.orchestra.run.smithi151.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T13:15:40.847 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T13:15:40.998 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T13:15:41.283 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T13:15:41.430 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T13:15:42.196 INFO:teuthology.orchestra.run.smithi098.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T13:15:42.201 DEBUG:teuthology.parallel:result is None 2024-09-16T13:15:44.070 INFO:teuthology.orchestra.run.smithi151.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T13:15:44.673 INFO:teuthology.orchestra.run.smithi151.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T13:15:44.677 DEBUG:teuthology.parallel:result is None 2024-09-16T13:15:44.678 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T13:15:44.833 DEBUG:teuthology.orchestra.run.smithi098:> dpkg-query -W -f '${Version}' ceph 2024-09-16T13:15:44.857 INFO:teuthology.orchestra.run.smithi098.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-16T13:15:44.857 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-16T13:15:44.858 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-16T13:15:44.859 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T13:15:45.008 DEBUG:teuthology.orchestra.run.smithi151:> dpkg-query -W -f '${Version}' ceph 2024-09-16T13:15:45.030 INFO:teuthology.orchestra.run.smithi151.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-16T13:15:45.031 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-16T13:15:45.031 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-16T13:15:45.033 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T13:15:45.033 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:15:45.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T13:15:45.050 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:15:45.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T13:15:45.065 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T13:15:45.065 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:15:45.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T13:15:45.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T13:15:45.117 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:15:45.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T13:15:45.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T13:15:45.143 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T13:15:45.143 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:15:45.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T13:15:45.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T13:15:45.214 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:15:45.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T13:15:45.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T13:15:45.288 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T13:15:45.288 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:15:45.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T13:15:45.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T13:15:45.319 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:15:45.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T13:15:45.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T13:15:45.346 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T13:15:45.423 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T13:15:45.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-09-16T13:15:45.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /var/log/ceph 2024-09-16T13:15:45.441 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T13:15:45.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T13:15:45.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T13:15:45.501 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T13:15:45.501 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T13:15:45.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T13:15:45.525 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T13:15:45.525 INFO:tasks.ceph:config {'conf': {'client': {'rbd cache': True, 'rbd cache policy': 'writeback'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T13:15:45.525 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_cache_writeback}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7907295', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'rbd cache': True, 'rbd cache policy': 'writeback'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi151.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'rbd_fsx': {'clients': ['client.0'], 'ops': 6000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_10:44:09', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T13:15:45.525 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T13:15:45.561 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T13:15:45.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T13:15:45.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T13:15:45.625 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:15:45.625 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T13:15:45.631 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T13:15:45.632 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-09-16T13:15:45.682 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T13:15:45.683 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:15:45.683 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 368 Links: 1 2024-09-16T13:15:45.683 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:15:45.683 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-16 13:12:17.330125023 +0000 2024-09-16T13:15:45.683 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-16 13:12:17.130129634 +0000 2024-09-16T13:15:45.683 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-16 13:12:17.130129634 +0000 2024-09-16T13:15:45.683 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-09-16T13:15:45.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T13:15:45.697 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-16T13:15:45.698 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-16T13:15:45.698 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000425527 s, 1.2 MB/s 2024-09-16T13:15:45.699 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T13:15:45.707 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-09-16T13:15:45.754 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T13:15:45.754 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:15:45.754 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 375 Links: 1 2024-09-16T13:15:45.754 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:15:45.754 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-16 13:12:17.558119768 +0000 2024-09-16T13:15:45.754 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-16 13:12:17.554119861 +0000 2024-09-16T13:15:45.755 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-16 13:12:17.554119861 +0000 2024-09-16T13:15:45.755 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-09-16T13:15:45.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T13:15:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-16T13:15:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-16T13:15:45.768 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000396918 s, 1.3 MB/s 2024-09-16T13:15:45.769 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T13:15:45.776 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-09-16T13:15:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T13:15:45.830 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:15:45.830 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 383 Links: 1 2024-09-16T13:15:45.830 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:15:45.830 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-16 13:12:17.966110364 +0000 2024-09-16T13:15:45.830 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-16 13:12:17.962110456 +0000 2024-09-16T13:15:45.831 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-16 13:12:17.962110456 +0000 2024-09-16T13:15:45.831 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-09-16T13:15:45.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T13:15:45.849 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-16T13:15:45.849 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-16T13:15:45.849 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000395979 s, 1.3 MB/s 2024-09-16T13:15:45.849 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T13:15:45.857 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-09-16T13:15:45.906 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T13:15:45.906 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:15:45.906 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 389 Links: 1 2024-09-16T13:15:45.906 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:15:45.906 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-16 13:12:18.374100960 +0000 2024-09-16T13:15:45.906 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-16 13:12:18.370101051 +0000 2024-09-16T13:15:45.906 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-16 13:12:18.370101051 +0000 2024-09-16T13:15:45.906 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-09-16T13:15:45.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T13:15:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-16T13:15:45.918 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-16T13:15:45.919 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000372818 s, 1.4 MB/s 2024-09-16T13:15:45.919 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T13:15:45.927 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T13:15:45.927 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:15:45.927 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T13:15:45.934 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T13:15:45.934 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-09-16T13:15:45.942 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T13:15:45.942 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:15:45.942 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 370 Links: 1 2024-09-16T13:15:45.942 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:15:45.942 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-16 13:12:20.895447187 +0000 2024-09-16T13:15:45.942 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-16 13:12:20.683452075 +0000 2024-09-16T13:15:45.942 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-16 13:12:20.683452075 +0000 2024-09-16T13:15:45.942 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2024-09-16T13:15:45.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T13:15:45.999 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-16T13:15:45.999 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-16T13:15:46.000 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000379855 s, 1.3 MB/s 2024-09-16T13:15:46.001 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T13:15:46.009 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-09-16T13:15:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T13:15:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:15:46.015 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-09-16T13:15:46.015 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:15:46.016 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-16 13:12:21.307437690 +0000 2024-09-16T13:15:46.016 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-16 13:12:21.099442485 +0000 2024-09-16T13:15:46.016 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-16 13:12:21.099442485 +0000 2024-09-16T13:15:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2024-09-16T13:15:46.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T13:15:46.071 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-16T13:15:46.071 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-16T13:15:46.071 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000403421 s, 1.3 MB/s 2024-09-16T13:15:46.073 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T13:15:46.081 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-09-16T13:15:46.087 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T13:15:46.087 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:15:46.087 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 383 Links: 1 2024-09-16T13:15:46.088 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:15:46.088 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-16 13:12:21.719428194 +0000 2024-09-16T13:15:46.088 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-16 13:12:21.511432988 +0000 2024-09-16T13:15:46.088 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-16 13:12:21.511432988 +0000 2024-09-16T13:15:46.088 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2024-09-16T13:15:46.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T13:15:46.101 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-16T13:15:46.101 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-16T13:15:46.102 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000400791 s, 1.3 MB/s 2024-09-16T13:15:46.102 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T13:15:46.110 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-09-16T13:15:46.160 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T13:15:46.160 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T13:15:46.160 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 6h/6d Inode: 389 Links: 1 2024-09-16T13:15:46.160 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T13:15:46.160 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-16 13:12:22.163417961 +0000 2024-09-16T13:15:46.160 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-16 13:12:21.955422754 +0000 2024-09-16T13:15:46.160 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-16 13:12:21.955422754 +0000 2024-09-16T13:15:46.160 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: - 2024-09-16T13:15:46.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T13:15:46.175 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-16T13:15:46.175 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-16T13:15:46.175 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000397221 s, 1.3 MB/s 2024-09-16T13:15:46.176 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T13:15:46.185 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T13:15:46.185 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T13:15:46.185 INFO:tasks.ceph:Generating config... 2024-09-16T13:15:46.186 INFO:tasks.ceph:[client] rbd cache = True 2024-09-16T13:15:46.186 INFO:tasks.ceph:[client] rbd cache policy = writeback 2024-09-16T13:15:46.186 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T13:15:46.187 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T13:15:46.187 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T13:15:46.187 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T13:15:46.187 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T13:15:46.187 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T13:15:46.187 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T13:15:46.187 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T13:15:46.187 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T13:15:46.187 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-16T13:15:46.187 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T13:15:46.187 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T13:15:46.187 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T13:15:46.187 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T13:15:46.187 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T13:15:46.188 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T13:15:46.188 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T13:15:46.188 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T13:15:46.188 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T13:15:46.188 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T13:15:46.188 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T13:15:46.188 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-16T13:15:46.188 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T13:15:46.188 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T13:15:46.188 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T13:15:46.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T13:15:46.223 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T13:15:46.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T13:15:46.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T13:15:46.275 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.98'), ('mon.c', '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]'), ('mon.b', '172.21.15.151')] 2024-09-16T13:15:46.276 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790],172.21.15.151', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd cache': True, 'rbd cache policy': 'writeback'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T13:15:46.276 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:15:46.277 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T13:15:46.326 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.98 --addv c '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]' --add b 172.21.15.151 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:15:46.365 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:15:46.365 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid 6ddcae89-6c18-4eb7-8712-1c53f3b07278 2024-09-16T13:15:46.365 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = octopus 2024-09-16T13:15:46.365 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2024-09-16T13:15:46.365 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 6ddcae89-6c18-4eb7-8712-1c53f3b07278 2024-09-16T13:15:46.365 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2024-09-16T13:15:46.362265+0000 2024-09-16T13:15:46.366 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-09-16T13:15:46.362265+0000 2024-09-16T13:15:46.366 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 15 (octopus) 2024-09-16T13:15:46.366 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 1 2024-09-16T13:15:46.366 INFO:teuthology.orchestra.run.smithi098.stdout:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-16T13:15:46.366 INFO:teuthology.orchestra.run.smithi098.stdout:1: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.b 2024-09-16T13:15:46.366 INFO:teuthology.orchestra.run.smithi098.stdout:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.c 2024-09-16T13:15:46.366 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T13:15:46.367 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T13:15:46.372 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6ddcae89-6c18-4eb7-8712-1c53f3b07278... 2024-09-16T13:15:46.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T13:15:46.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T13:15:46.403 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-09-16T13:15:46.403 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-09-16T13:15:46.403 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T13:15:46.403 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-09-16T13:15:46.403 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.403 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-09-16T13:15:46.403 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T13:15:46.404 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-09-16T13:15:46.405 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790],172.21.15.151" 2024-09-16T13:15:46.406 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on pool no app = False 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 5000 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 6ddcae89-6c18-4eb7-8712-1c53f3b07278 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-09-16T13:15:46.407 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-09-16T13:15:46.408 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression algorithm = snappy 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression mode = aggressive 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 1/20 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 1/20 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 4/10 2024-09-16T13:15:46.409 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: osd max markdown count = 1000 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: osd shutdown pgref assert = True 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-09-16T13:15:46.410 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-09-16T13:15:46.411 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.412 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout: rbd cache = True 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout: rbd cache policy = writeback 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-09-16T13:15:46.413 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout: chdir = "" 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout: auth supported = cephx 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout: filestore xattr use omap = true 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout: mon clock drift allowed = 1.000 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout: osd crush chooseleaf type = 0 2024-09-16T13:15:46.414 INFO:teuthology.orchestra.run.smithi151.stdout: auth debug = true 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on old message = true 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: ms die on bug = true 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: mon pg warn max object skew = 0 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default size = 2 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.415 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow primary affinity = true 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd allow pg remap = true 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on legacy crush tunables = false 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on crush straw calc version zero = false 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on no sortbitwise = false 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on osd down out interval zero = false 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on too few osds = false 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: mon_allow_pool_size_one = true 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: osd default data pool replay window = 5 2024-09-16T13:15:46.416 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: mon allow pool delete = true 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: mon cluster log file level = debug 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: debug asserts on shutdown = true 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: mon health detail to clog = false 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: mon host = "172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790],172.21.15.151" 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: mon client directed command retry = 5 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: mon warn on pool no app = False 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: ms inject socket failures = 5000 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 6ddcae89-6c18-4eb7-8712-1c53f3b07278 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout:[osd] 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: osd journal size = 100 2024-09-16T13:15:46.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub load threshold = 5.0 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: osd scrub max interval = 600 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: osd mclock profile = high_recovery_ops 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: osd recover clone overlap = true 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: osd recovery max chunk = 1048576 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug shutdown = true 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug op order = true 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug verify stray on activate = true 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: osd open classes on start = true 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug pg log writeout = true 2024-09-16T13:15:46.418 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: osd deep scrub update digest min age = 30 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: osd map max advance = 10 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: journal zero on create = true 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: filestore ondisk finisher threads = 3 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: filestore apply finisher threads = 3 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: bdev debug aio = true 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: osd debug misdirected ops = true 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore block size = 96636764160 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore compression algorithm = snappy 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore compression mode = aggressive 2024-09-16T13:15:46.419 INFO:teuthology.orchestra.run.smithi151.stdout: bluestore fsck on mount = True 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluefs = 1/20 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: debug bluestore = 1/20 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: debug osd = 20 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: debug rocksdb = 4/10 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd full ratio = 0.9 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd nearfull ratio = 0.8 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: osd failsafe full ratio = 0.95 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: osd max markdown count = 1000 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: osd objectstore = bluestore 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: osd shutdown pgref assert = True 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.420 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr] 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: debug mgr = 20 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: mgr/telemetry/nag = false 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout:[mon] 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: debug ms = 1 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: debug mon = 20 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: debug paxos = 20 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: debug auth = 20 2024-09-16T13:15:46.421 INFO:teuthology.orchestra.run.smithi151.stdout: mon data avail warn = 5 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: mon mgr mkfs grace = 240 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min pgs per osd = 4 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd reporter subtree level = osd 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: mon osd prime pg temp = true 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: mon reweight min bytes per osd = 10 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T13:15:46.422 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: # 1m isn't quite enough 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: mon_down_mkfs_grace = 2m 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: mon_warn_on_filestore_osds = false 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout:[client] 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: rgw cache enabled = true 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable ops log = true 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: rgw enable usage log = true 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: rbd cache = True 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout: rbd cache policy = writeback 2024-09-16T13:15:46.423 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.a] 2024-09-16T13:15:46.424 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.c] 2024-09-16T13:15:46.424 INFO:teuthology.orchestra.run.smithi151.stdout:[mon.b] 2024-09-16T13:15:46.425 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T13:15:46.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T13:15:46.474 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T13:15:46.474 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:15:46.474 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T13:15:46.481 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:15:46.481 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T13:15:46.486 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:15:46.486 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:15:46.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T13:15:46.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T13:15:46.511 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:15:46.511 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:15:46.517 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:15:46.517 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:15:46.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T13:15:46.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T13:15:46.544 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:15:46.545 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:15:46.591 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T13:15:46.591 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T13:15:46.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T13:15:46.637 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T13:15:46.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T13:15:46.686 INFO:teuthology.orchestra.run.smithi151.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T13:15:46.688 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T13:15:46.689 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T13:15:46.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T13:15:46.723 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T13:15:46.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-16T13:15:46.770 INFO:teuthology.orchestra.run.smithi151.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-16T13:15:46.783 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T13:15:46.783 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi151.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T13:15:46.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T13:15:46.800 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T13:15:46.800 INFO:tasks.ceph:role: osd.0 2024-09-16T13:15:46.800 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:15:46.800 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T13:15:47.314 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:15:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:15:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:15:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 2024-09-16T13:15:47.314 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:15:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:15:47.314 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:15:47.315 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T13:15:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:15:47.315 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:15:47.316 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-09-16T13:15:47.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T13:15:47.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T13:15:47.429 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T13:15:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:15:47.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T13:15:47.440 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T13:15:47.440 INFO:tasks.ceph:role: osd.1 2024-09-16T13:15:47.440 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:15:47.440 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T13:15:48.018 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:15:48.018 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:15:48.018 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:15:48.018 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 2024-09-16T13:15:48.018 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:15:48.018 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:15:48.019 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:15:48.019 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T13:15:48.019 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:15:48.019 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:15:48.020 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-09-16T13:15:48.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T13:15:48.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T13:15:48.064 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T13:15:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:15:48.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T13:15:48.075 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T13:15:48.076 INFO:tasks.ceph:role: osd.2 2024-09-16T13:15:48.076 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:15:48.076 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T13:15:48.598 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:15:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:15:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:15:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 2024-09-16T13:15:48.599 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:15:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:15:48.599 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:15:48.599 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T13:15:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:15:48.599 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:15:48.600 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-09-16T13:15:48.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T13:15:48.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T13:15:48.676 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T13:15:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:15:48.678 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T13:15:48.687 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T13:15:48.688 INFO:tasks.ceph:role: osd.3 2024-09-16T13:15:48.688 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:15:48.688 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T13:15:49.191 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:15:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:15:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:15:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 2024-09-16T13:15:49.191 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:15:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:15:49.191 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:15:49.191 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T13:15:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:15:49.191 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:15:49.192 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-09-16T13:15:49.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-16T13:15:49.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T13:15:49.273 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T13:15:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:15:49.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:15:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:49.338+0000 7fa588201540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T13:15:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:49.338+0000 7fa588201540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T13:15:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:49.338+0000 7fa588201540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T13:15:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:49.338+0000 7fa588201540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T13:15:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:49.338+0000 7fa588201540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T13:15:49.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:49.342+0000 7fa588201540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T13:15:49.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:49.902+0000 7fa588201540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:15:50.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:50.682+0000 7fa588201540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:15:51.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:51.469+0000 7fa588201540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:15:52.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T13:15:52.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:15:52.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:52.397+0000 7f88daf96540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T13:15:52.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:52.397+0000 7f88daf96540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T13:15:52.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:52.397+0000 7f88daf96540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T13:15:52.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:52.397+0000 7f88daf96540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T13:15:52.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:52.397+0000 7f88daf96540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T13:15:52.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:52.397+0000 7f88daf96540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T13:15:52.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:52.953+0000 7f88daf96540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:15:53.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:53.737+0000 7f88daf96540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:15:54.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:54.517+0000 7f88daf96540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:15:55.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T13:15:55.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:15:55.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:55.469+0000 7f1c3f8c7540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T13:15:55.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:55.469+0000 7f1c3f8c7540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T13:15:55.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:55.469+0000 7f1c3f8c7540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T13:15:55.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:55.469+0000 7f1c3f8c7540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T13:15:55.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:55.469+0000 7f1c3f8c7540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T13:15:55.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:55.469+0000 7f1c3f8c7540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T13:15:56.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:56.029+0000 7f1c3f8c7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:15:56.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:56.825+0000 7f1c3f8c7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:15:57.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:57.605+0000 7f1c3f8c7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:15:58.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T13:15:58.492 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:15:58.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:58.565+0000 7f2358919540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T13:15:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:58.565+0000 7f2358919540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T13:15:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:58.565+0000 7f2358919540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T13:15:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:58.565+0000 7f2358919540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T13:15:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:58.565+0000 7f2358919540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T13:15:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:58.565+0000 7f2358919540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T13:15:59.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:59.117+0000 7f2358919540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:15:59.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:15:59.901+0000 7f2358919540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:16:00.681+0000 7f2358919540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:01.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T13:16:01.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T13:16:01.572 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T13:16:01.573 INFO:tasks.ceph:role: osd.4 2024-09-16T13:16:01.573 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:16:01.573 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T13:16:02.109 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:16:02.109 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:16:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:16:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2024-09-16T13:16:02.110 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:16:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:16:02.110 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:16:02.110 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T13:16:02.110 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:16:02.110 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:16:02.111 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-09-16T13:16:02.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-16T13:16:02.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T13:16:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:16:02.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T13:16:02.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T13:16:02.213 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T13:16:02.213 INFO:tasks.ceph:role: osd.5 2024-09-16T13:16:02.214 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:16:02.214 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T13:16:02.752 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:16:02.918 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:16:02.918 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:16:02.918 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2024-09-16T13:16:02.919 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:16:02.919 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:16:02.919 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:16:02.919 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T13:16:02.919 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:16:02.919 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:16:02.921 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-09-16T13:16:02.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T13:16:02.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T13:16:02.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T13:16:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:16:02.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T13:16:02.969 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T13:16:02.969 INFO:tasks.ceph:role: osd.6 2024-09-16T13:16:02.969 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:16:02.970 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T13:16:03.456 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:16:03.457 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:16:03.457 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:16:03.457 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2024-09-16T13:16:03.457 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:16:03.457 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:16:03.457 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:16:03.457 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T13:16:03.457 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:16:03.457 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:16:03.458 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-09-16T13:16:03.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-16T13:16:03.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T13:16:03.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T13:16:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:16:03.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T13:16:03.547 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T13:16:03.547 INFO:tasks.ceph:role: osd.7 2024-09-16T13:16:03.547 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:16:03.547 DEBUG:teuthology.orchestra.run.smithi151:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T13:16:04.043 INFO:teuthology.orchestra.run.smithi151.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T13:16:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T13:16:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T13:16:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: = reflink=1 2024-09-16T13:16:04.043 INFO:teuthology.orchestra.run.smithi151.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T13:16:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: = sunit=0 swidth=0 blks 2024-09-16T13:16:04.043 INFO:teuthology.orchestra.run.smithi151.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T13:16:04.043 INFO:teuthology.orchestra.run.smithi151.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T13:16:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T13:16:04.043 INFO:teuthology.orchestra.run.smithi151.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T13:16:04.044 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi151.front.sepia.ceph.com -o noatime 2024-09-16T13:16:04.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-16T13:16:04.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T13:16:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:16:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T13:16:04.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:16:04.217 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:04.208+0000 7fd172485540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T13:16:04.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:04.212+0000 7fd172485540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T13:16:04.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:04.212+0000 7fd172485540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T13:16:04.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:04.212+0000 7fd172485540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T13:16:04.220 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:04.212+0000 7fd172485540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T13:16:04.224 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:04.216+0000 7fd172485540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T13:16:04.787 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:04.780+0000 7fd172485540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:05.571 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:05.564+0000 7fd172485540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:06.353 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:06.344+0000 7fd172485540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:07.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T13:16:07.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:16:07.303 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:07.296+0000 7f505e802540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T13:16:07.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:07.296+0000 7f505e802540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T13:16:07.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:07.296+0000 7f505e802540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T13:16:07.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:07.296+0000 7f505e802540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T13:16:07.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:07.296+0000 7f505e802540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T13:16:07.304 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:07.296+0000 7f505e802540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T13:16:07.859 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:07.852+0000 7f505e802540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:08.650 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:08.643+0000 7f505e802540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:09.445 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:09.439+0000 7f505e802540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:10.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T13:16:10.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:16:10.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:10.351+0000 7f45a871a540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T13:16:10.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:10.351+0000 7f45a871a540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T13:16:10.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:10.351+0000 7f45a871a540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-16T13:16:10.361 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:10.351+0000 7f45a871a540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-16T13:16:10.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:10.351+0000 7f45a871a540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-16T13:16:10.362 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:10.355+0000 7f45a871a540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T13:16:10.913 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:10.903+0000 7f45a871a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:11.697 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:11.687+0000 7f45a871a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:12.478 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:12.471+0000 7f45a871a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:13.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T13:16:13.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:16:13.438 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:13.431+0000 7fe284d80540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T13:16:13.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:13.431+0000 7fe284d80540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T13:16:13.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:13.431+0000 7fe284d80540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-16T13:16:13.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:13.431+0000 7fe284d80540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-16T13:16:13.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:13.431+0000 7fe284d80540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-16T13:16:13.439 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:13.431+0000 7fe284d80540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T13:16:13.997 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:13.987+0000 7fe284d80540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:14.788 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:14.779+0000 7fe284d80540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:15.569 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:16:15.559+0000 7fe284d80540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T13:16:16.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T13:16:16.424 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T13:16:16.425 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:16:16.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T13:16:16.442 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:16:16.442 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T13:16:16.498 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:16:16.499 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T13:16:16.508 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:16:16.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T13:16:16.519 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:16:16.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T13:16:16.529 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:16:16.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T13:16:16.546 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:16:16.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T13:16:16.556 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:16:16.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T13:16:16.572 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:16:16.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T13:16:16.629 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:16:16.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T13:16:16.640 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:16:16.640 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T13:16:16.647 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:16:16.647 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-16T13:16:16.652 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T13:16:16.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T13:16:16.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T13:16:16.695 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCCL+hmlxHSJRAATelVoT85w2elkgrqbj9RZA== 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCFL+hmSb1rFBAABbT1wdbz7QQEAz4fqRZ9Zg== 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCIL+hm4K0HGBAAKYV9vAoXByr91cW/+QHJZw== 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCLL+hmr641HBAAR4XdQmDlBbGEHp8ff4/eiA== 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCOL+hmwv/+IRAAo2kQZZCqoFPHgP3XroyLKQ== 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-09-16T13:16:16.696 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCCL+hmJQW9KBAA5yRFO3Opx+pd+xq3g/j16g== 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCUL+hmRuTFDBAA1znbCT1mK7M8NSQZwMQC9g== 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCXL+hmLD7tERAA7YRdPMOYgikmT1p07SfLDA== 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCaL+hm2cVXFRAAuDxe/RBExGDlK+PG5mwG6A== 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCdL+hm/Yn0GRAAjN1A1BQwzGGqMKFtEkCPww== 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCCL+hmZ3DwKhAAg8ub+uVlQIQwaSOHOxtGvA== 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2024-09-16T13:16:16.697 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCCL+hmpKrGLRAAIRNUbktFri8aAgmz0FJIQA== 2024-09-16T13:16:16.705 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.y] 2024-09-16T13:16:16.705 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCCL+hmlxHSJRAATelVoT85w2elkgrqbj9RZA== 2024-09-16T13:16:16.705 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.0] 2024-09-16T13:16:16.705 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCFL+hmSb1rFBAABbT1wdbz7QQEAz4fqRZ9Zg== 2024-09-16T13:16:16.705 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.1] 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCIL+hm4K0HGBAAKYV9vAoXByr91cW/+QHJZw== 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.2] 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCLL+hmr641HBAAR4XdQmDlBbGEHp8ff4/eiA== 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.3] 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCOL+hmwv/+IRAAo2kQZZCqoFPHgP3XroyLKQ== 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout:[mgr.x] 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCCL+hmJQW9KBAA5yRFO3Opx+pd+xq3g/j16g== 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.4] 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCUL+hmRuTFDBAA1znbCT1mK7M8NSQZwMQC9g== 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.5] 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCXL+hmLD7tERAA7YRdPMOYgikmT1p07SfLDA== 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.6] 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCaL+hm2cVXFRAAuDxe/RBExGDlK+PG5mwG6A== 2024-09-16T13:16:16.706 INFO:teuthology.orchestra.run.smithi151.stdout:[osd.7] 2024-09-16T13:16:16.707 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCdL+hm/Yn0GRAAjN1A1BQwzGGqMKFtEkCPww== 2024-09-16T13:16:16.707 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-09-16T13:16:16.707 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCCL+hmZ3DwKhAAg8ub+uVlQIQwaSOHOxtGvA== 2024-09-16T13:16:16.707 INFO:teuthology.orchestra.run.smithi151.stdout:[client.1] 2024-09-16T13:16:16.707 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQCCL+hmpKrGLRAAIRNUbktFri8aAgmz0FJIQA== 2024-09-16T13:16:16.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T13:16:16.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T13:16:16.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:16.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:16.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:16.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:16.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:16.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:16.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:16.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:17.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T13:16:17.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T13:16:17.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:17.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:17.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:17.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:17.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:17.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:17.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:17.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T13:16:17.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T13:16:17.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T13:16:17.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T13:16:17.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T13:16:17.469 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T13:16:17.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T13:16:17.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T13:16:17.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T13:16:17.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T13:16:17.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T13:16:17.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T13:16:17.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T13:16:17.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T13:16:17.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T13:16:17.796 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:16:17.798 DEBUG:teuthology.orchestra.run.smithi151:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T13:16:17.803 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T13:16:17.803 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T13:16:17.803 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T13:16:17.845 INFO:tasks.ceph.mon.a:Started 2024-09-16T13:16:17.845 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T13:16:17.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T13:16:17.847 INFO:tasks.ceph.mon.c:Started 2024-09-16T13:16:17.848 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T13:16:17.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T13:16:17.850 INFO:tasks.ceph.mon.b:Started 2024-09-16T13:16:17.850 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T13:16:17.850 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T13:16:17.850 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T13:16:17.853 INFO:tasks.ceph.mgr.y:Started 2024-09-16T13:16:17.853 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T13:16:17.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T13:16:17.855 INFO:tasks.ceph.mgr.x:Started 2024-09-16T13:16:17.855 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T13:16:17.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T13:16:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-09-16T13:16:23.299 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T13:16:23.299 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:16:23.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T13:16:23.315 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:16:23.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T13:16:23.330 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:16:23.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T13:16:23.343 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T13:16:23.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T13:16:23.391 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:23.381+0000 7fda500d6000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T13:16:23.397 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:16:23.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T13:16:23.413 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:16:23.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T13:16:23.419 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:23.411+0000 7fb7fe313000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T13:16:23.434 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:16:23.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T13:16:23.490 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-16T13:16:23.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T13:16:23.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new be8543b4-3d28-4b37-a4ce-d2a56a09aeaf 0 2024-09-16T13:16:23.707 INFO:teuthology.orchestra.run.smithi151.stdout:0 2024-09-16T13:16:23.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new a7f95c7b-56d4-437b-a160-cabb8fd2b1e4 1 2024-09-16T13:16:23.912 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-09-16T13:16:23.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c 2 2024-09-16T13:16:24.130 INFO:teuthology.orchestra.run.smithi151.stdout:2 2024-09-16T13:16:24.134 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:24.127+0000 7fb7fe313000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T13:16:24.137 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:24.129+0000 7fda500d6000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T13:16:24.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 4898b5b5-b912-430c-adcb-c52d90226917 3 2024-09-16T13:16:24.194 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:24.187+0000 7fb7fe313000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T13:16:24.194 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:24.185+0000 7fda500d6000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T13:16:24.310 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:24.301+0000 7fda500d6000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T13:16:24.324 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:24.315+0000 7fb7fe313000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T13:16:24.360 INFO:teuthology.orchestra.run.smithi151.stdout:3 2024-09-16T13:16:24.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 6c478be2-f606-41d8-967c-9a689f09a440 4 2024-09-16T13:16:24.550 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:24.541+0000 7fda500d6000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T13:16:24.558 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:24.547+0000 7fb7fe313000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T13:16:24.571 INFO:teuthology.orchestra.run.smithi151.stdout:4 2024-09-16T13:16:24.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 5001c828-b223-4d2c-b893-f9c56da73b83 5 2024-09-16T13:16:24.603 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:24.593+0000 7fda500d6000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T13:16:24.609 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:24.599+0000 7fb7fe313000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T13:16:24.777 INFO:teuthology.orchestra.run.smithi151.stdout:5 2024-09-16T13:16:24.784 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:24.775+0000 7fb7fe313000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T13:16:24.791 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:24.781+0000 7fda500d6000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T13:16:24.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new 6f7a9125-674c-4be2-93e1-b6296cd00f8e 6 2024-09-16T13:16:24.845 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:24.835+0000 7fb7fe313000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T13:16:24.856 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:24.849+0000 7fda500d6000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T13:16:24.906 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:24.899+0000 7fb7fe313000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T13:16:24.923 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:24.913+0000 7fda500d6000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T13:16:24.996 INFO:teuthology.orchestra.run.smithi151.stdout:6 2024-09-16T13:16:25.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo ceph --cluster ceph osd new bcdfe435-f24d-45a9-a629-7380ad2e575a 7 2024-09-16T13:16:25.210 INFO:teuthology.orchestra.run.smithi151.stdout:7 2024-09-16T13:16:25.225 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T13:16:25.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T13:16:25.227 INFO:tasks.ceph.osd.0:Started 2024-09-16T13:16:25.227 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T13:16:25.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T13:16:25.230 INFO:tasks.ceph.osd.1:Started 2024-09-16T13:16:25.230 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T13:16:25.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T13:16:25.232 INFO:tasks.ceph.osd.2:Started 2024-09-16T13:16:25.232 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T13:16:25.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T13:16:25.234 INFO:tasks.ceph.osd.3:Started 2024-09-16T13:16:25.235 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T13:16:25.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T13:16:25.237 INFO:tasks.ceph.osd.4:Started 2024-09-16T13:16:25.237 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T13:16:25.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T13:16:25.239 INFO:tasks.ceph.osd.5:Started 2024-09-16T13:16:25.239 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T13:16:25.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T13:16:25.241 INFO:tasks.ceph.osd.6:Started 2024-09-16T13:16:25.241 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T13:16:25.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T13:16:25.242 INFO:tasks.ceph.osd.7:Started 2024-09-16T13:16:25.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:16:25.330 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:25.323+0000 7fb7fe313000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T13:16:25.417 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:25.409+0000 7fda500d6000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T13:16:25.441 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:25.431+0000 7fb7fe313000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T13:16:25.463 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:16:25.463 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:16:25.201328+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T13:16:25.201328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:16:25.474 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T13:16:25.474 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T13:16:25.538 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:25.529+0000 7fda500d6000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T13:16:25.580 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:25.571+0000 7f01e7f79540 -1 Falling back to public interface 2024-09-16T13:16:25.594 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:25.587+0000 7fb0dfa8e540 -1 Falling back to public interface 2024-09-16T13:16:25.622 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:25.613+0000 7fa45b14f540 -1 Falling back to public interface 2024-09-16T13:16:25.627 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:25.619+0000 7fa06a572540 -1 Falling back to public interface 2024-09-16T13:16:25.633 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:25.625+0000 7f2bc595f540 -1 Falling back to public interface 2024-09-16T13:16:25.634 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:25.625+0000 7f5405982540 -1 Falling back to public interface 2024-09-16T13:16:25.735 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:25.727+0000 7fb7fe313000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T13:16:25.853 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:25.845+0000 7f8731e1b540 -1 Falling back to public interface 2024-09-16T13:16:25.859 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:25.849+0000 7fda500d6000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T13:16:25.871 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:25.863+0000 7f672cef6540 -1 Falling back to public interface 2024-09-16T13:16:25.901 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:25.891+0000 7fb7fe313000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T13:16:25.967 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:25.959+0000 7fb7fe313000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T13:16:26.016 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:26.007+0000 7fb7fe313000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T13:16:26.021 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:26.013+0000 7fda500d6000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T13:16:26.072 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:26.063+0000 7fb7fe313000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T13:16:26.087 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:26.077+0000 7fda500d6000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T13:16:26.139 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:26.129+0000 7fda500d6000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T13:16:26.197 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:26.189+0000 7fda500d6000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T13:16:26.234 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:26.227+0000 7fb7fe313000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T13:16:26.280 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T13:16:26.360 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:26.353+0000 7fda500d6000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T13:16:26.402 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:26.391+0000 7fb7fe313000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T13:16:26.450 INFO:teuthology.misc.health.smithi098.stdout: 2024-09-16T13:16:26.450 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:16:25.201328+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T13:16:25.201328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:16:26.460 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-16T13:16:26.467 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:26.459+0000 7fb7fe313000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T13:16:26.524 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:26.517+0000 7fda500d6000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T13:16:26.591 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:26.581+0000 7fda500d6000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T13:16:26.626 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:26.619+0000 7fb7fe313000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T13:16:26.694 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:26.687+0000 7fb7fe313000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T13:16:26.749 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:26.741+0000 7fda500d6000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T13:16:26.818 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:26.809+0000 7fda500d6000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T13:16:27.405 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:27.395+0000 7fb7fe313000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T13:16:27.533 INFO:tasks.ceph.mgr.y.smithi098.stderr:2024-09-16T13:16:27.525+0000 7fda500d6000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T13:16:28.452 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:28.443+0000 7fb7e738a700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T13:16:28.452 INFO:tasks.ceph.mgr.x.smithi151.stderr:2024-09-16T13:16:28.443+0000 7fb7e738a700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T13:16:29.214 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:29.205+0000 7f2bc595f540 -1 osd.0 0 log_to_monitors true 2024-09-16T13:16:29.229 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:29.223+0000 7fb0dfa8e540 -1 osd.5 0 log_to_monitors true 2024-09-16T13:16:29.389 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:29.379+0000 7f01e7f79540 -1 osd.7 0 log_to_monitors true 2024-09-16T13:16:29.483 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:29.475+0000 7fa06a572540 -1 osd.6 0 log_to_monitors true 2024-09-16T13:16:29.515 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:29.505+0000 7f5405982540 -1 osd.1 0 log_to_monitors true 2024-09-16T13:16:29.521 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:29.513+0000 7fa45b14f540 -1 osd.2 0 log_to_monitors true 2024-09-16T13:16:29.637 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:29.627+0000 7f672cef6540 -1 osd.4 0 log_to_monitors true 2024-09-16T13:16:29.648 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:29.641+0000 7f8731e1b540 -1 osd.3 0 log_to_monitors true 2024-09-16T13:16:31.766 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:31.757+0000 7f2bc2924700 -1 osd.0 0 waiting for initial osdmap 2024-09-16T13:16:31.775 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:31.765+0000 7f2bb9ea2700 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:16:31.836 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:31.827+0000 7fb0dca53700 -1 osd.5 0 waiting for initial osdmap 2024-09-16T13:16:31.841 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:31.831+0000 7f01e4f3e700 -1 osd.7 0 waiting for initial osdmap 2024-09-16T13:16:31.842 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:31.835+0000 7fb0d3fd1700 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:16:31.850 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:31.843+0000 7f01dc4bc700 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:16:32.756 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:32.745+0000 7fa458114700 -1 osd.2 0 waiting for initial osdmap 2024-09-16T13:16:32.761 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:32.753+0000 7fa44f692700 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:16:32.767 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:32.759+0000 7fa065d12700 -1 osd.6 0 waiting for initial osdmap 2024-09-16T13:16:32.771 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:32.763+0000 7fa05eab5700 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:16:32.790 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:32.781+0000 7f872d5bb700 -1 osd.3 0 waiting for initial osdmap 2024-09-16T13:16:32.794 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:32.787+0000 7f6728696700 -1 osd.4 0 waiting for initial osdmap 2024-09-16T13:16:32.796 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:32.789+0000 7f872635e700 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:16:32.799 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:32.791+0000 7f6721439700 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:16:32.810 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:32.801+0000 7f5401122700 -1 osd.1 0 waiting for initial osdmap 2024-09-16T13:16:32.816 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:32.809+0000 7f53f9ec5700 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:16:33.265 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T13:16:33.555 INFO:teuthology.misc.health.smithi098.stdout: 2024-09-16T13:16:33.555 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":14,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:16:32.763994+0000","last_up_change":"2024-09-16T13:16:32.763994+0000","last_in_change":"2024-09-16T13:16:25.201328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":22271},{"type":"v1","addr":"172.21.15.98:6809","nonce":22271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":22271},{"type":"v1","addr":"172.21.15.98:6811","nonce":22271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":22271},{"type":"v1","addr":"172.21.15.98:6817","nonce":22271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":22271},{"type":"v1","addr":"172.21.15.98:6814","nonce":22271}]},"public_addr":"172.21.15.98:6809/22271","cluster_addr":"172.21.15.98:6811/22271","heartbeat_back_addr":"172.21.15.98:6817/22271","heartbeat_front_addr":"172.21.15.98:6814/22271","state":["exists","up"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":22273},{"type":"v1","addr":"172.21.15.98:6801","nonce":22273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":22273},{"type":"v1","addr":"172.21.15.98:6803","nonce":22273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":22273},{"type":"v1","addr":"172.21.15.98:6807","nonce":22273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":22273},{"type":"v1","addr":"172.21.15.98:6805","nonce":22273}]},"public_addr":"172.21.15.98:6801/22273","cluster_addr":"172.21.15.98:6803/22273","heartbeat_back_addr":"172.21.15.98:6807/22273","heartbeat_front_addr":"172.21.15.98:6805/22273","state":["exists","up"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":22285},{"type":"v1","addr":"172.21.15.151:6809","nonce":22285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":22285},{"type":"v1","addr":"172.21.15.151:6811","nonce":22285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":22285},{"type":"v1","addr":"172.21.15.151:6815","nonce":22285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":22285},{"type":"v1","addr":"172.21.15.151:6813","nonce":22285}]},"public_addr":"172.21.15.151:6809/22285","cluster_addr":"172.21.15.151:6811/22285","heartbeat_back_addr":"172.21.15.151:6815/22285","heartbeat_front_addr":"172.21.15.151:6813/22285","state":["exists","up"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":22284},{"type":"v1","addr":"172.21.15.151:6801","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":22284},{"type":"v1","addr":"172.21.15.151:6803","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":22284},{"type":"v1","addr":"172.21.15.151:6807","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":22284},{"type":"v1","addr":"172.21.15.151:6805","nonce":22284}]},"public_addr":"172.21.15.151:6801/22284","cluster_addr":"172.21.15.151:6803/22284","heartbeat_back_addr":"172.21.15.151:6807/22284","heartbeat_front_addr":"172.21.15.151:6805/22284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:16:33.567 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-09-16T13:16:40.374 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T13:16:40.659 INFO:teuthology.misc.health.smithi098.stdout: 2024-09-16T13:16:40.659 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":17,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:16:35.771381+0000","last_up_change":"2024-09-16T13:16:33.765413+0000","last_in_change":"2024-09-16T13:16:25.201328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:16:33.497561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":22271},{"type":"v1","addr":"172.21.15.98:6809","nonce":22271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":22271},{"type":"v1","addr":"172.21.15.98:6811","nonce":22271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":22271},{"type":"v1","addr":"172.21.15.98:6817","nonce":22271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":22271},{"type":"v1","addr":"172.21.15.98:6814","nonce":22271}]},"public_addr":"172.21.15.98:6809/22271","cluster_addr":"172.21.15.98:6811/22271","heartbeat_back_addr":"172.21.15.98:6817/22271","heartbeat_front_addr":"172.21.15.98:6814/22271","state":["exists","up"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":22272},{"type":"v1","addr":"172.21.15.98:6815","nonce":22272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":22272},{"type":"v1","addr":"172.21.15.98:6819","nonce":22272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":22272},{"type":"v1","addr":"172.21.15.98:6823","nonce":22272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":22272},{"type":"v1","addr":"172.21.15.98:6821","nonce":22272}]},"public_addr":"172.21.15.98:6815/22272","cluster_addr":"172.21.15.98:6819/22272","heartbeat_back_addr":"172.21.15.98:6823/22272","heartbeat_front_addr":"172.21.15.98:6821/22272","state":["exists","up"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":22273},{"type":"v1","addr":"172.21.15.98:6801","nonce":22273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":22273},{"type":"v1","addr":"172.21.15.98:6803","nonce":22273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":22273},{"type":"v1","addr":"172.21.15.98:6807","nonce":22273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":22273},{"type":"v1","addr":"172.21.15.98:6805","nonce":22273}]},"public_addr":"172.21.15.98:6801/22273","cluster_addr":"172.21.15.98:6803/22273","heartbeat_back_addr":"172.21.15.98:6807/22273","heartbeat_front_addr":"172.21.15.98:6805/22273","state":["exists","up"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":22270},{"type":"v1","addr":"172.21.15.98:6825","nonce":22270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":22270},{"type":"v1","addr":"172.21.15.98:6827","nonce":22270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":22270},{"type":"v1","addr":"172.21.15.98:6831","nonce":22270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":22270},{"type":"v1","addr":"172.21.15.98:6829","nonce":22270}]},"public_addr":"172.21.15.98:6825/22270","cluster_addr":"172.21.15.98:6827/22270","heartbeat_back_addr":"172.21.15.98:6831/22270","heartbeat_front_addr":"172.21.15.98:6829/22270","state":["exists","up"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":22286},{"type":"v1","addr":"172.21.15.151:6825","nonce":22286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":22286},{"type":"v1","addr":"172.21.15.151:6827","nonce":22286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":22286},{"type":"v1","addr":"172.21.15.151:6831","nonce":22286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":22286},{"type":"v1","addr":"172.21.15.151:6829","nonce":22286}]},"public_addr":"172.21.15.151:6825/22286","cluster_addr":"172.21.15.151:6827/22286","heartbeat_back_addr":"172.21.15.151:6831/22286","heartbeat_front_addr":"172.21.15.151:6829/22286","state":["exists","up"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":22285},{"type":"v1","addr":"172.21.15.151:6809","nonce":22285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":22285},{"type":"v1","addr":"172.21.15.151:6811","nonce":22285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":22285},{"type":"v1","addr":"172.21.15.151:6815","nonce":22285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":22285},{"type":"v1","addr":"172.21.15.151:6813","nonce":22285}]},"public_addr":"172.21.15.151:6809/22285","cluster_addr":"172.21.15.151:6811/22285","heartbeat_back_addr":"172.21.15.151:6815/22285","heartbeat_front_addr":"172.21.15.151:6813/22285","state":["exists","up"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":22315},{"type":"v1","addr":"172.21.15.151:6817","nonce":22315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":22315},{"type":"v1","addr":"172.21.15.151:6819","nonce":22315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":22315},{"type":"v1","addr":"172.21.15.151:6823","nonce":22315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":22315},{"type":"v1","addr":"172.21.15.151:6821","nonce":22315}]},"public_addr":"172.21.15.151:6817/22315","cluster_addr":"172.21.15.151:6819/22315","heartbeat_back_addr":"172.21.15.151:6823/22315","heartbeat_front_addr":"172.21.15.151:6821/22315","state":["exists","up"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":22284},{"type":"v1","addr":"172.21.15.151:6801","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":22284},{"type":"v1","addr":"172.21.15.151:6803","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":22284},{"type":"v1","addr":"172.21.15.151:6807","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":22284},{"type":"v1","addr":"172.21.15.151:6805","nonce":22284}]},"public_addr":"172.21.15.151:6801/22284","cluster_addr":"172.21.15.151:6803/22284","heartbeat_back_addr":"172.21.15.151:6807/22284","heartbeat_front_addr":"172.21.15.151:6805/22284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.191311+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.563551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.493457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.599815+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.634352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.262873+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.508881+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.348276+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:16:40.672 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-16T13:16:40.672 INFO:tasks.ceph:Creating RBD pool 2024-09-16T13:16:40.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T13:16:41.819 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'rbd' created 2024-09-16T13:16:41.831 DEBUG:teuthology.orchestra.run.smithi098:> rbd --cluster ceph pool init rbd 2024-09-16T13:16:44.853 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T13:16:44.854 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T13:16:44.854 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T13:16:44.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T13:16:44.854 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T13:16:45.197 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:16:45.210 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":21987},{"type":"v1","addr":"172.21.15.151:6833","nonce":21987}]},"active_addr":"172.21.15.151:6833/21987","active_change":"2024-09-16T13:16:27.421699+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":2791626159}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":684825360}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":2781857361}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":3411258758}]}]} 2024-09-16T13:16:45.214 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T13:16:45.214 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T13:16:45.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:16:45.500 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:16:45.500 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:16:44.821317+0000","last_up_change":"2024-09-16T13:16:33.765413+0000","last_in_change":"2024-09-16T13:16:25.201328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:16:33.497561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:16:40.994538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":22271},{"type":"v1","addr":"172.21.15.98:6809","nonce":22271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":22271},{"type":"v1","addr":"172.21.15.98:6811","nonce":22271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":22271},{"type":"v1","addr":"172.21.15.98:6817","nonce":22271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":22271},{"type":"v1","addr":"172.21.15.98:6814","nonce":22271}]},"public_addr":"172.21.15.98:6809/22271","cluster_addr":"172.21.15.98:6811/22271","heartbeat_back_addr":"172.21.15.98:6817/22271","heartbeat_front_addr":"172.21.15.98:6814/22271","state":["exists","up"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":22272},{"type":"v1","addr":"172.21.15.98:6815","nonce":22272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":22272},{"type":"v1","addr":"172.21.15.98:6819","nonce":22272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":22272},{"type":"v1","addr":"172.21.15.98:6823","nonce":22272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":22272},{"type":"v1","addr":"172.21.15.98:6821","nonce":22272}]},"public_addr":"172.21.15.98:6815/22272","cluster_addr":"172.21.15.98:6819/22272","heartbeat_back_addr":"172.21.15.98:6823/22272","heartbeat_front_addr":"172.21.15.98:6821/22272","state":["exists","up"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":22273},{"type":"v1","addr":"172.21.15.98:6801","nonce":22273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":22273},{"type":"v1","addr":"172.21.15.98:6803","nonce":22273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":22273},{"type":"v1","addr":"172.21.15.98:6807","nonce":22273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":22273},{"type":"v1","addr":"172.21.15.98:6805","nonce":22273}]},"public_addr":"172.21.15.98:6801/22273","cluster_addr":"172.21.15.98:6803/22273","heartbeat_back_addr":"172.21.15.98:6807/22273","heartbeat_front_addr":"172.21.15.98:6805/22273","state":["exists","up"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":22270},{"type":"v1","addr":"172.21.15.98:6825","nonce":22270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":22270},{"type":"v1","addr":"172.21.15.98:6827","nonce":22270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":22270},{"type":"v1","addr":"172.21.15.98:6831","nonce":22270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":22270},{"type":"v1","addr":"172.21.15.98:6829","nonce":22270}]},"public_addr":"172.21.15.98:6825/22270","cluster_addr":"172.21.15.98:6827/22270","heartbeat_back_addr":"172.21.15.98:6831/22270","heartbeat_front_addr":"172.21.15.98:6829/22270","state":["exists","up"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":22286},{"type":"v1","addr":"172.21.15.151:6825","nonce":22286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":22286},{"type":"v1","addr":"172.21.15.151:6827","nonce":22286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":22286},{"type":"v1","addr":"172.21.15.151:6831","nonce":22286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":22286},{"type":"v1","addr":"172.21.15.151:6829","nonce":22286}]},"public_addr":"172.21.15.151:6825/22286","cluster_addr":"172.21.15.151:6827/22286","heartbeat_back_addr":"172.21.15.151:6831/22286","heartbeat_front_addr":"172.21.15.151:6829/22286","state":["exists","up"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":22285},{"type":"v1","addr":"172.21.15.151:6809","nonce":22285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":22285},{"type":"v1","addr":"172.21.15.151:6811","nonce":22285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":22285},{"type":"v1","addr":"172.21.15.151:6815","nonce":22285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":22285},{"type":"v1","addr":"172.21.15.151:6813","nonce":22285}]},"public_addr":"172.21.15.151:6809/22285","cluster_addr":"172.21.15.151:6811/22285","heartbeat_back_addr":"172.21.15.151:6815/22285","heartbeat_front_addr":"172.21.15.151:6813/22285","state":["exists","up"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":22315},{"type":"v1","addr":"172.21.15.151:6817","nonce":22315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":22315},{"type":"v1","addr":"172.21.15.151:6819","nonce":22315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":22315},{"type":"v1","addr":"172.21.15.151:6823","nonce":22315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":22315},{"type":"v1","addr":"172.21.15.151:6821","nonce":22315}]},"public_addr":"172.21.15.151:6817/22315","cluster_addr":"172.21.15.151:6819/22315","heartbeat_back_addr":"172.21.15.151:6823/22315","heartbeat_front_addr":"172.21.15.151:6821/22315","state":["exists","up"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":22284},{"type":"v1","addr":"172.21.15.151:6801","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":22284},{"type":"v1","addr":"172.21.15.151:6803","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":22284},{"type":"v1","addr":"172.21.15.151:6807","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":22284},{"type":"v1","addr":"172.21.15.151:6805","nonce":22284}]},"public_addr":"172.21.15.151:6801/22284","cluster_addr":"172.21.15.151:6803/22284","heartbeat_back_addr":"172.21.15.151:6807/22284","heartbeat_front_addr":"172.21.15.151:6805/22284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.191311+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.563551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.493457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.599815+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.634352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.262873+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.508881+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.348276+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:16:45.514 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T13:16:45.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:16:45.804 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:16:45.805 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:16:44.821317+0000","last_up_change":"2024-09-16T13:16:33.765413+0000","last_in_change":"2024-09-16T13:16:25.201328+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:16:33.497561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:16:40.994538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":22271},{"type":"v1","addr":"172.21.15.98:6809","nonce":22271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":22271},{"type":"v1","addr":"172.21.15.98:6811","nonce":22271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":22271},{"type":"v1","addr":"172.21.15.98:6817","nonce":22271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":22271},{"type":"v1","addr":"172.21.15.98:6814","nonce":22271}]},"public_addr":"172.21.15.98:6809/22271","cluster_addr":"172.21.15.98:6811/22271","heartbeat_back_addr":"172.21.15.98:6817/22271","heartbeat_front_addr":"172.21.15.98:6814/22271","state":["exists","up"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":22272},{"type":"v1","addr":"172.21.15.98:6815","nonce":22272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":22272},{"type":"v1","addr":"172.21.15.98:6819","nonce":22272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":22272},{"type":"v1","addr":"172.21.15.98:6823","nonce":22272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":22272},{"type":"v1","addr":"172.21.15.98:6821","nonce":22272}]},"public_addr":"172.21.15.98:6815/22272","cluster_addr":"172.21.15.98:6819/22272","heartbeat_back_addr":"172.21.15.98:6823/22272","heartbeat_front_addr":"172.21.15.98:6821/22272","state":["exists","up"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":22273},{"type":"v1","addr":"172.21.15.98:6801","nonce":22273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":22273},{"type":"v1","addr":"172.21.15.98:6803","nonce":22273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":22273},{"type":"v1","addr":"172.21.15.98:6807","nonce":22273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":22273},{"type":"v1","addr":"172.21.15.98:6805","nonce":22273}]},"public_addr":"172.21.15.98:6801/22273","cluster_addr":"172.21.15.98:6803/22273","heartbeat_back_addr":"172.21.15.98:6807/22273","heartbeat_front_addr":"172.21.15.98:6805/22273","state":["exists","up"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":22270},{"type":"v1","addr":"172.21.15.98:6825","nonce":22270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":22270},{"type":"v1","addr":"172.21.15.98:6827","nonce":22270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":22270},{"type":"v1","addr":"172.21.15.98:6831","nonce":22270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":22270},{"type":"v1","addr":"172.21.15.98:6829","nonce":22270}]},"public_addr":"172.21.15.98:6825/22270","cluster_addr":"172.21.15.98:6827/22270","heartbeat_back_addr":"172.21.15.98:6831/22270","heartbeat_front_addr":"172.21.15.98:6829/22270","state":["exists","up"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":22286},{"type":"v1","addr":"172.21.15.151:6825","nonce":22286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":22286},{"type":"v1","addr":"172.21.15.151:6827","nonce":22286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":22286},{"type":"v1","addr":"172.21.15.151:6831","nonce":22286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":22286},{"type":"v1","addr":"172.21.15.151:6829","nonce":22286}]},"public_addr":"172.21.15.151:6825/22286","cluster_addr":"172.21.15.151:6827/22286","heartbeat_back_addr":"172.21.15.151:6831/22286","heartbeat_front_addr":"172.21.15.151:6829/22286","state":["exists","up"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":22285},{"type":"v1","addr":"172.21.15.151:6809","nonce":22285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":22285},{"type":"v1","addr":"172.21.15.151:6811","nonce":22285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":22285},{"type":"v1","addr":"172.21.15.151:6815","nonce":22285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":22285},{"type":"v1","addr":"172.21.15.151:6813","nonce":22285}]},"public_addr":"172.21.15.151:6809/22285","cluster_addr":"172.21.15.151:6811/22285","heartbeat_back_addr":"172.21.15.151:6815/22285","heartbeat_front_addr":"172.21.15.151:6813/22285","state":["exists","up"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":22315},{"type":"v1","addr":"172.21.15.151:6817","nonce":22315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":22315},{"type":"v1","addr":"172.21.15.151:6819","nonce":22315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":22315},{"type":"v1","addr":"172.21.15.151:6823","nonce":22315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":22315},{"type":"v1","addr":"172.21.15.151:6821","nonce":22315}]},"public_addr":"172.21.15.151:6817/22315","cluster_addr":"172.21.15.151:6819/22315","heartbeat_back_addr":"172.21.15.151:6823/22315","heartbeat_front_addr":"172.21.15.151:6821/22315","state":["exists","up"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":22284},{"type":"v1","addr":"172.21.15.151:6801","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":22284},{"type":"v1","addr":"172.21.15.151:6803","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":22284},{"type":"v1","addr":"172.21.15.151:6807","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":22284},{"type":"v1","addr":"172.21.15.151:6805","nonce":22284}]},"public_addr":"172.21.15.151:6801/22284","cluster_addr":"172.21.15.151:6803/22284","heartbeat_back_addr":"172.21.15.151:6807/22284","heartbeat_front_addr":"172.21.15.151:6805/22284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.191311+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.563551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.493457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.599815+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.634352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.262873+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.508881+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.348276+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:16:45.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T13:16:45.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T13:16:45.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T13:16:45.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T13:16:45.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T13:16:45.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T13:16:45.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T13:16:45.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T13:16:46.022 INFO:teuthology.orchestra.run.smithi098.stdout:60129542149 2024-09-16T13:16:46.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T13:16:46.035 INFO:teuthology.orchestra.run.smithi098.stdout:60129542148 2024-09-16T13:16:46.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T13:16:46.036 INFO:teuthology.orchestra.run.smithi098.stdout:64424509445 2024-09-16T13:16:46.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T13:16:46.037 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2024-09-16T13:16:46.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T13:16:46.041 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2024-09-16T13:16:46.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T13:16:46.043 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2024-09-16T13:16:46.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T13:16:46.049 INFO:teuthology.orchestra.run.smithi098.stdout:60129542149 2024-09-16T13:16:46.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T13:16:46.051 INFO:teuthology.orchestra.run.smithi098.stdout:60129542149 2024-09-16T13:16:46.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T13:16:46.498 INFO:teuthology.orchestra.run.smithi098.stdout:64424509444 2024-09-16T13:16:46.507 INFO:teuthology.orchestra.run.smithi098.stdout:60129542148 2024-09-16T13:16:46.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.4 2024-09-16T13:16:46.513 INFO:teuthology.orchestra.run.smithi098.stdout:60129542147 2024-09-16T13:16:46.514 INFO:teuthology.orchestra.run.smithi098.stdout:64424509445 2024-09-16T13:16:46.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.0 2024-09-16T13:16:46.525 INFO:teuthology.orchestra.run.smithi098.stdout:64424509445 2024-09-16T13:16:46.529 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.2 2024-09-16T13:16:46.533 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.6 2024-09-16T13:16:46.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.3 2024-09-16T13:16:46.542 INFO:teuthology.orchestra.run.smithi098.stdout:64424509445 2024-09-16T13:16:46.548 INFO:teuthology.orchestra.run.smithi098.stdout:60129542148 2024-09-16T13:16:46.548 INFO:teuthology.orchestra.run.smithi098.stdout:60129542148 2024-09-16T13:16:46.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.1 2024-09-16T13:16:46.560 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.5 2024-09-16T13:16:46.565 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.7 2024-09-16T13:16:47.512 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T13:16:47.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T13:16:47.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T13:16:47.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T13:16:47.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T13:16:47.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T13:16:47.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T13:16:47.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T13:16:47.945 INFO:teuthology.orchestra.run.smithi098.stdout:64424509445 2024-09-16T13:16:47.964 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.4 2024-09-16T13:16:47.964 DEBUG:teuthology.parallel:result is None 2024-09-16T13:16:47.973 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2024-09-16T13:16:47.989 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.6 2024-09-16T13:16:47.990 DEBUG:teuthology.parallel:result is None 2024-09-16T13:16:48.009 INFO:teuthology.orchestra.run.smithi098.stdout:60129542149 2024-09-16T13:16:48.011 INFO:teuthology.orchestra.run.smithi098.stdout:60129542148 2024-09-16T13:16:48.019 INFO:teuthology.orchestra.run.smithi098.stdout:60129542149 2024-09-16T13:16:48.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.0 2024-09-16T13:16:48.024 DEBUG:teuthology.parallel:result is None 2024-09-16T13:16:48.027 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2024-09-16T13:16:48.029 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.2 2024-09-16T13:16:48.029 DEBUG:teuthology.parallel:result is None 2024-09-16T13:16:48.032 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.7 2024-09-16T13:16:48.032 DEBUG:teuthology.parallel:result is None 2024-09-16T13:16:48.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.3 2024-09-16T13:16:48.041 DEBUG:teuthology.parallel:result is None 2024-09-16T13:16:48.043 INFO:teuthology.orchestra.run.smithi098.stdout:60129542149 2024-09-16T13:16:48.050 INFO:teuthology.orchestra.run.smithi098.stdout:64424509446 2024-09-16T13:16:48.054 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.5 2024-09-16T13:16:48.055 DEBUG:teuthology.parallel:result is None 2024-09-16T13:16:48.065 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.1 2024-09-16T13:16:48.065 DEBUG:teuthology.parallel:result is None 2024-09-16T13:16:48.065 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T13:16:48.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:16:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:16:48.387 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:16:48.398 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T13:16:47.934702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2388960,"kb_used_data":1768,"kb_used_omap":13,"kb_used_meta":2387058,"kb_avail":752585760,"statfs":{"total":773094113280,"available":770647818240,"internally_reserved":0,"allocated":1810432,"data_stored":1744638,"data_compressed":65840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":13632,"internal_metadata":2444348096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.119728"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.343674+0000","last_change":"2024-09-16T13:16:45.344339+0000","last_active":"2024-09-16T13:16:45.343674+0000","last_peered":"2024-09-16T13:16:45.343674+0000","last_clean":"2024-09-16T13:16:45.343674+0000","last_became_active":"2024-09-16T13:16:42.825708+0000","last_became_peered":"2024-09-16T13:16:42.825708+0000","last_unstale":"2024-09-16T13:16:45.343674+0000","last_undegraded":"2024-09-16T13:16:45.343674+0000","last_fullsized":"2024-09-16T13:16:45.343674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:23.678862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.845059+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:16:44.845059+0000","last_peered":"2024-09-16T13:16:44.845059+0000","last_clean":"2024-09-16T13:16:44.845059+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:16:44.845059+0000","last_undegraded":"2024-09-16T13:16:44.845059+0000","last_fullsized":"2024-09-16T13:16:44.845059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.344488+0000","last_change":"2024-09-16T13:16:45.344887+0000","last_active":"2024-09-16T13:16:45.344488+0000","last_peered":"2024-09-16T13:16:45.344488+0000","last_clean":"2024-09-16T13:16:45.344488+0000","last_became_active":"2024-09-16T13:16:42.821357+0000","last_became_peered":"2024-09-16T13:16:42.821357+0000","last_unstale":"2024-09-16T13:16:45.344488+0000","last_undegraded":"2024-09-16T13:16:45.344488+0000","last_fullsized":"2024-09-16T13:16:45.344488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:35.373375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.844949+0000","last_change":"2024-09-16T13:16:44.845588+0000","last_active":"2024-09-16T13:16:44.844949+0000","last_peered":"2024-09-16T13:16:44.844949+0000","last_clean":"2024-09-16T13:16:44.844949+0000","last_became_active":"2024-09-16T13:16:42.823703+0000","last_became_peered":"2024-09-16T13:16:42.823703+0000","last_unstale":"2024-09-16T13:16:44.844949+0000","last_undegraded":"2024-09-16T13:16:44.844949+0000","last_fullsized":"2024-09-16T13:16:44.844949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:02:15.873787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.843137+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:16:44.843137+0000","last_peered":"2024-09-16T13:16:44.843137+0000","last_clean":"2024-09-16T13:16:44.843137+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:16:44.843137+0000","last_undegraded":"2024-09-16T13:16:44.843137+0000","last_fullsized":"2024-09-16T13:16:44.843137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.128433+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:16:45.128433+0000","last_peered":"2024-09-16T13:16:45.128433+0000","last_clean":"2024-09-16T13:16:45.128433+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:16:45.128433+0000","last_undegraded":"2024-09-16T13:16:45.128433+0000","last_fullsized":"2024-09-16T13:16:45.128433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.344378+0000","last_change":"2024-09-16T13:16:45.344856+0000","last_active":"2024-09-16T13:16:45.344378+0000","last_peered":"2024-09-16T13:16:45.344378+0000","last_clean":"2024-09-16T13:16:45.344378+0000","last_became_active":"2024-09-16T13:16:42.825917+0000","last_became_peered":"2024-09-16T13:16:42.825917+0000","last_unstale":"2024-09-16T13:16:45.344378+0000","last_undegraded":"2024-09-16T13:16:45.344378+0000","last_fullsized":"2024-09-16T13:16:45.344378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:30.133715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082018499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.840901+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:16:44.840901+0000","last_peered":"2024-09-16T13:16:44.840901+0000","last_clean":"2024-09-16T13:16:44.840901+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:16:44.840901+0000","last_undegraded":"2024-09-16T13:16:44.840901+0000","last_fullsized":"2024-09-16T13:16:44.840901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'72","reported_seq":103,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T13:16:43.829089+0000","last_change":"2024-09-16T13:16:34.788814+0000","last_active":"2024-09-16T13:16:43.829089+0000","last_peered":"2024-09-16T13:16:43.829089+0000","last_clean":"2024-09-16T13:16:43.829089+0000","last_became_active":"2024-09-16T13:16:34.786680+0000","last_became_peered":"2024-09-16T13:16:34.786680+0000","last_unstale":"2024-09-16T13:16:43.829089+0000","last_undegraded":"2024-09-16T13:16:43.829089+0000","last_fullsized":"2024-09-16T13:16:43.829089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:17.345314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7930000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":212992,"data_stored":70502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301792,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070048,"statfs":{"total":96636764160,"available":96327729152,"internally_reserved":0,"allocated":274432,"data_stored":660851,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297568,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074272,"statfs":{"total":96636764160,"available":96332054528,"internally_reserved":0,"allocated":212992,"data_stored":70502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6910000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8460000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301788,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070052,"statfs":{"total":96636764160,"available":96327733248,"internally_reserved":0,"allocated":274432,"data_stored":660851,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:16:48.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:16:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:16:48.672 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:16:48.682 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T13:16:47.934702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2388960,"kb_used_data":1768,"kb_used_omap":13,"kb_used_meta":2387058,"kb_avail":752585760,"statfs":{"total":773094113280,"available":770647818240,"internally_reserved":0,"allocated":1810432,"data_stored":1744638,"data_compressed":65840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":13632,"internal_metadata":2444348096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.119728"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.343674+0000","last_change":"2024-09-16T13:16:45.344339+0000","last_active":"2024-09-16T13:16:45.343674+0000","last_peered":"2024-09-16T13:16:45.343674+0000","last_clean":"2024-09-16T13:16:45.343674+0000","last_became_active":"2024-09-16T13:16:42.825708+0000","last_became_peered":"2024-09-16T13:16:42.825708+0000","last_unstale":"2024-09-16T13:16:45.343674+0000","last_undegraded":"2024-09-16T13:16:45.343674+0000","last_fullsized":"2024-09-16T13:16:45.343674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:23.678862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.845059+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:16:44.845059+0000","last_peered":"2024-09-16T13:16:44.845059+0000","last_clean":"2024-09-16T13:16:44.845059+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:16:44.845059+0000","last_undegraded":"2024-09-16T13:16:44.845059+0000","last_fullsized":"2024-09-16T13:16:44.845059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.344488+0000","last_change":"2024-09-16T13:16:45.344887+0000","last_active":"2024-09-16T13:16:45.344488+0000","last_peered":"2024-09-16T13:16:45.344488+0000","last_clean":"2024-09-16T13:16:45.344488+0000","last_became_active":"2024-09-16T13:16:42.821357+0000","last_became_peered":"2024-09-16T13:16:42.821357+0000","last_unstale":"2024-09-16T13:16:45.344488+0000","last_undegraded":"2024-09-16T13:16:45.344488+0000","last_fullsized":"2024-09-16T13:16:45.344488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:35.373375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.844949+0000","last_change":"2024-09-16T13:16:44.845588+0000","last_active":"2024-09-16T13:16:44.844949+0000","last_peered":"2024-09-16T13:16:44.844949+0000","last_clean":"2024-09-16T13:16:44.844949+0000","last_became_active":"2024-09-16T13:16:42.823703+0000","last_became_peered":"2024-09-16T13:16:42.823703+0000","last_unstale":"2024-09-16T13:16:44.844949+0000","last_undegraded":"2024-09-16T13:16:44.844949+0000","last_fullsized":"2024-09-16T13:16:44.844949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:02:15.873787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.843137+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:16:44.843137+0000","last_peered":"2024-09-16T13:16:44.843137+0000","last_clean":"2024-09-16T13:16:44.843137+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:16:44.843137+0000","last_undegraded":"2024-09-16T13:16:44.843137+0000","last_fullsized":"2024-09-16T13:16:44.843137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.128433+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:16:45.128433+0000","last_peered":"2024-09-16T13:16:45.128433+0000","last_clean":"2024-09-16T13:16:45.128433+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:16:45.128433+0000","last_undegraded":"2024-09-16T13:16:45.128433+0000","last_fullsized":"2024-09-16T13:16:45.128433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.344378+0000","last_change":"2024-09-16T13:16:45.344856+0000","last_active":"2024-09-16T13:16:45.344378+0000","last_peered":"2024-09-16T13:16:45.344378+0000","last_clean":"2024-09-16T13:16:45.344378+0000","last_became_active":"2024-09-16T13:16:42.825917+0000","last_became_peered":"2024-09-16T13:16:42.825917+0000","last_unstale":"2024-09-16T13:16:45.344378+0000","last_undegraded":"2024-09-16T13:16:45.344378+0000","last_fullsized":"2024-09-16T13:16:45.344378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:30.133715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082018499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.840901+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:16:44.840901+0000","last_peered":"2024-09-16T13:16:44.840901+0000","last_clean":"2024-09-16T13:16:44.840901+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:16:44.840901+0000","last_undegraded":"2024-09-16T13:16:44.840901+0000","last_fullsized":"2024-09-16T13:16:44.840901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'72","reported_seq":103,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T13:16:43.829089+0000","last_change":"2024-09-16T13:16:34.788814+0000","last_active":"2024-09-16T13:16:43.829089+0000","last_peered":"2024-09-16T13:16:43.829089+0000","last_clean":"2024-09-16T13:16:43.829089+0000","last_became_active":"2024-09-16T13:16:34.786680+0000","last_became_peered":"2024-09-16T13:16:34.786680+0000","last_unstale":"2024-09-16T13:16:43.829089+0000","last_undegraded":"2024-09-16T13:16:43.829089+0000","last_fullsized":"2024-09-16T13:16:43.829089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:17.345314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7930000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":212992,"data_stored":70502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301792,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070048,"statfs":{"total":96636764160,"available":96327729152,"internally_reserved":0,"allocated":274432,"data_stored":660851,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297568,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074272,"statfs":{"total":96636764160,"available":96332054528,"internally_reserved":0,"allocated":212992,"data_stored":70502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6910000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8460000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301788,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070052,"statfs":{"total":96636764160,"available":96327733248,"internally_reserved":0,"allocated":274432,"data_stored":660851,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:16:48.683 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T13:16:48.683 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T13:16:48.683 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T13:16:48.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T13:16:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:16:48.972 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T13:16:48.984 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T13:16:48.984 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T13:16:48.996 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T13:16:48.997 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'rbd cache': True, 'rbd cache policy': 'writeback'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-16T13:16:48.997 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-16T13:16:48.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-16T13:16:49.289 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T13:16:49.289 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T13:16:49.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:16:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:16:49.569 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:16:49.580 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T13:16:49.440190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2388968,"kb_used_data":1780,"kb_used_omap":13,"kb_used_meta":2387058,"kb_avail":752585752,"statfs":{"total":773094113280,"available":770647810048,"internally_reserved":0,"allocated":1822720,"data_stored":1750092,"data_compressed":65840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":13632,"internal_metadata":2444348096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001745"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.343674+0000","last_change":"2024-09-16T13:16:45.344339+0000","last_active":"2024-09-16T13:16:45.343674+0000","last_peered":"2024-09-16T13:16:45.343674+0000","last_clean":"2024-09-16T13:16:45.343674+0000","last_became_active":"2024-09-16T13:16:42.825708+0000","last_became_peered":"2024-09-16T13:16:42.825708+0000","last_unstale":"2024-09-16T13:16:45.343674+0000","last_undegraded":"2024-09-16T13:16:45.343674+0000","last_fullsized":"2024-09-16T13:16:45.343674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:23.678862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.845059+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:16:44.845059+0000","last_peered":"2024-09-16T13:16:44.845059+0000","last_clean":"2024-09-16T13:16:44.845059+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:16:44.845059+0000","last_undegraded":"2024-09-16T13:16:44.845059+0000","last_fullsized":"2024-09-16T13:16:44.845059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.344488+0000","last_change":"2024-09-16T13:16:45.344887+0000","last_active":"2024-09-16T13:16:45.344488+0000","last_peered":"2024-09-16T13:16:45.344488+0000","last_clean":"2024-09-16T13:16:45.344488+0000","last_became_active":"2024-09-16T13:16:42.821357+0000","last_became_peered":"2024-09-16T13:16:42.821357+0000","last_unstale":"2024-09-16T13:16:45.344488+0000","last_undegraded":"2024-09-16T13:16:45.344488+0000","last_fullsized":"2024-09-16T13:16:45.344488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:35.373375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.844949+0000","last_change":"2024-09-16T13:16:44.845588+0000","last_active":"2024-09-16T13:16:44.844949+0000","last_peered":"2024-09-16T13:16:44.844949+0000","last_clean":"2024-09-16T13:16:44.844949+0000","last_became_active":"2024-09-16T13:16:42.823703+0000","last_became_peered":"2024-09-16T13:16:42.823703+0000","last_unstale":"2024-09-16T13:16:44.844949+0000","last_undegraded":"2024-09-16T13:16:44.844949+0000","last_fullsized":"2024-09-16T13:16:44.844949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:02:15.873787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.843137+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:16:44.843137+0000","last_peered":"2024-09-16T13:16:44.843137+0000","last_clean":"2024-09-16T13:16:44.843137+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:16:44.843137+0000","last_undegraded":"2024-09-16T13:16:44.843137+0000","last_fullsized":"2024-09-16T13:16:44.843137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.128433+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:16:45.128433+0000","last_peered":"2024-09-16T13:16:45.128433+0000","last_clean":"2024-09-16T13:16:45.128433+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:16:45.128433+0000","last_undegraded":"2024-09-16T13:16:45.128433+0000","last_fullsized":"2024-09-16T13:16:45.128433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.344378+0000","last_change":"2024-09-16T13:16:45.344856+0000","last_active":"2024-09-16T13:16:45.344378+0000","last_peered":"2024-09-16T13:16:45.344378+0000","last_clean":"2024-09-16T13:16:45.344378+0000","last_became_active":"2024-09-16T13:16:42.825917+0000","last_became_peered":"2024-09-16T13:16:42.825917+0000","last_unstale":"2024-09-16T13:16:45.344378+0000","last_undegraded":"2024-09-16T13:16:45.344378+0000","last_fullsized":"2024-09-16T13:16:45.344378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:30.133715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082018499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.840901+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:16:44.840901+0000","last_peered":"2024-09-16T13:16:44.840901+0000","last_clean":"2024-09-16T13:16:44.840901+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:16:44.840901+0000","last_undegraded":"2024-09-16T13:16:44.840901+0000","last_fullsized":"2024-09-16T13:16:44.840901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'72","reported_seq":103,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T13:16:43.829089+0000","last_change":"2024-09-16T13:16:34.788814+0000","last_active":"2024-09-16T13:16:43.829089+0000","last_peered":"2024-09-16T13:16:43.829089+0000","last_clean":"2024-09-16T13:16:43.829089+0000","last_became_active":"2024-09-16T13:16:34.786680+0000","last_became_peered":"2024-09-16T13:16:34.786680+0000","last_unstale":"2024-09-16T13:16:43.829089+0000","last_undegraded":"2024-09-16T13:16:43.829089+0000","last_fullsized":"2024-09-16T13:16:43.829089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:17.345314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7930000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":212992,"data_stored":70502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301792,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070048,"statfs":{"total":96636764160,"available":96327729152,"internally_reserved":0,"allocated":274432,"data_stored":660851,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297568,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074272,"statfs":{"total":96636764160,"available":96332054528,"internally_reserved":0,"allocated":212992,"data_stored":70502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301796,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070044,"statfs":{"total":96636764160,"available":96327725056,"internally_reserved":0,"allocated":286720,"data_stored":666305,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:16:49.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:16:49.872 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:16:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:16:49.888 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T13:16:49.440190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2388968,"kb_used_data":1780,"kb_used_omap":13,"kb_used_meta":2387058,"kb_avail":752585752,"statfs":{"total":773094113280,"available":770647810048,"internally_reserved":0,"allocated":1822720,"data_stored":1750092,"data_compressed":65840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":13632,"internal_metadata":2444348096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001745"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.343674+0000","last_change":"2024-09-16T13:16:45.344339+0000","last_active":"2024-09-16T13:16:45.343674+0000","last_peered":"2024-09-16T13:16:45.343674+0000","last_clean":"2024-09-16T13:16:45.343674+0000","last_became_active":"2024-09-16T13:16:42.825708+0000","last_became_peered":"2024-09-16T13:16:42.825708+0000","last_unstale":"2024-09-16T13:16:45.343674+0000","last_undegraded":"2024-09-16T13:16:45.343674+0000","last_fullsized":"2024-09-16T13:16:45.343674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:23.678862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.845059+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:16:44.845059+0000","last_peered":"2024-09-16T13:16:44.845059+0000","last_clean":"2024-09-16T13:16:44.845059+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:16:44.845059+0000","last_undegraded":"2024-09-16T13:16:44.845059+0000","last_fullsized":"2024-09-16T13:16:44.845059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.344488+0000","last_change":"2024-09-16T13:16:45.344887+0000","last_active":"2024-09-16T13:16:45.344488+0000","last_peered":"2024-09-16T13:16:45.344488+0000","last_clean":"2024-09-16T13:16:45.344488+0000","last_became_active":"2024-09-16T13:16:42.821357+0000","last_became_peered":"2024-09-16T13:16:42.821357+0000","last_unstale":"2024-09-16T13:16:45.344488+0000","last_undegraded":"2024-09-16T13:16:45.344488+0000","last_fullsized":"2024-09-16T13:16:45.344488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:35.373375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.844949+0000","last_change":"2024-09-16T13:16:44.845588+0000","last_active":"2024-09-16T13:16:44.844949+0000","last_peered":"2024-09-16T13:16:44.844949+0000","last_clean":"2024-09-16T13:16:44.844949+0000","last_became_active":"2024-09-16T13:16:42.823703+0000","last_became_peered":"2024-09-16T13:16:42.823703+0000","last_unstale":"2024-09-16T13:16:44.844949+0000","last_undegraded":"2024-09-16T13:16:44.844949+0000","last_fullsized":"2024-09-16T13:16:44.844949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:02:15.873787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.843137+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:16:44.843137+0000","last_peered":"2024-09-16T13:16:44.843137+0000","last_clean":"2024-09-16T13:16:44.843137+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:16:44.843137+0000","last_undegraded":"2024-09-16T13:16:44.843137+0000","last_fullsized":"2024-09-16T13:16:44.843137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.128433+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:16:45.128433+0000","last_peered":"2024-09-16T13:16:45.128433+0000","last_clean":"2024-09-16T13:16:45.128433+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:16:45.128433+0000","last_undegraded":"2024-09-16T13:16:45.128433+0000","last_fullsized":"2024-09-16T13:16:45.128433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:45.344378+0000","last_change":"2024-09-16T13:16:45.344856+0000","last_active":"2024-09-16T13:16:45.344378+0000","last_peered":"2024-09-16T13:16:45.344378+0000","last_clean":"2024-09-16T13:16:45.344378+0000","last_became_active":"2024-09-16T13:16:42.825917+0000","last_became_peered":"2024-09-16T13:16:42.825917+0000","last_unstale":"2024-09-16T13:16:45.344378+0000","last_undegraded":"2024-09-16T13:16:45.344378+0000","last_fullsized":"2024-09-16T13:16:45.344378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:30.133715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082018499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"19'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T13:16:44.840901+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:16:44.840901+0000","last_peered":"2024-09-16T13:16:44.840901+0000","last_clean":"2024-09-16T13:16:44.840901+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:16:44.840901+0000","last_undegraded":"2024-09-16T13:16:44.840901+0000","last_fullsized":"2024-09-16T13:16:44.840901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"17'72","reported_seq":103,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T13:16:43.829089+0000","last_change":"2024-09-16T13:16:34.788814+0000","last_active":"2024-09-16T13:16:43.829089+0000","last_peered":"2024-09-16T13:16:43.829089+0000","last_clean":"2024-09-16T13:16:43.829089+0000","last_became_active":"2024-09-16T13:16:34.786680+0000","last_became_peered":"2024-09-16T13:16:34.786680+0000","last_unstale":"2024-09-16T13:16:43.829089+0000","last_undegraded":"2024-09-16T13:16:43.829089+0000","last_fullsized":"2024-09-16T13:16:43.829089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:17.345314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7930000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297564,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074276,"statfs":{"total":96636764160,"available":96332058624,"internally_reserved":0,"allocated":212992,"data_stored":70502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301792,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070048,"statfs":{"total":96636764160,"available":96327729152,"internally_reserved":0,"allocated":274432,"data_stored":660851,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297568,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074272,"statfs":{"total":96636764160,"available":96332054528,"internally_reserved":0,"allocated":212992,"data_stored":70502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297560,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074280,"statfs":{"total":96636764160,"available":96332062720,"internally_reserved":0,"allocated":208896,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301796,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070044,"statfs":{"total":96636764160,"available":96327725056,"internally_reserved":0,"allocated":286720,"data_stored":666305,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:16:49.889 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T13:16:49.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T13:16:50.195 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 22 2024-09-16T13:16:50.196 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 6ddcae89-6c18-4eb7-8712-1c53f3b07278 2024-09-16T13:16:50.196 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-09-16T13:16:23.176689+0000 2024-09-16T13:16:50.196 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-09-16T13:16:47.927392+0000 2024-09-16T13:16:50.196 INFO:teuthology.orchestra.run.smithi098.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T13:16:50.196 INFO:teuthology.orchestra.run.smithi098.stdout:crush_version 5 2024-09-16T13:16:50.196 INFO:teuthology.orchestra.run.smithi098.stdout:full_ratio 0.95 2024-09-16T13:16:50.196 INFO:teuthology.orchestra.run.smithi098.stdout:backfillfull_ratio 0.9 2024-09-16T13:16:50.196 INFO:teuthology.orchestra.run.smithi098.stdout:nearfull_ratio 0.85 2024-09-16T13:16:50.196 INFO:teuthology.orchestra.run.smithi098.stdout:require_min_compat_client luminous 2024-09-16T13:16:50.196 INFO:teuthology.orchestra.run.smithi098.stdout:min_compat_client jewel 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:require_osd_release quincy 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:stretch_mode_enabled false 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 21 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:max_osd 8 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6808/22271,v1:172.21.15.98:6809/22271] [v2:172.21.15.98:6810/22271,v1:172.21.15.98:6811/22271] exists,up be8543b4-3d28-4b37-a4ce-d2a56a09aeaf 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6812/22272,v1:172.21.15.98:6815/22272] [v2:172.21.15.98:6818/22272,v1:172.21.15.98:6819/22272] exists,up a7f95c7b-56d4-437b-a160-cabb8fd2b1e4 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 up in weight 1 up_from 14 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6800/22273,v1:172.21.15.98:6801/22273] [v2:172.21.15.98:6802/22273,v1:172.21.15.98:6803/22273] exists,up a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6824/22270,v1:172.21.15.98:6825/22270] [v2:172.21.15.98:6826/22270,v1:172.21.15.98:6827/22270] exists,up 4898b5b5-b912-430c-adcb-c52d90226917 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6824/22286,v1:172.21.15.151:6825/22286] [v2:172.21.15.151:6826/22286,v1:172.21.15.151:6827/22286] exists,up 6c478be2-f606-41d8-967c-9a689f09a440 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 up in weight 1 up_from 14 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6808/22285,v1:172.21.15.151:6809/22285] [v2:172.21.15.151:6810/22285,v1:172.21.15.151:6811/22285] exists,up 5001c828-b223-4d2c-b893-f9c56da73b83 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6816/22315,v1:172.21.15.151:6817/22315] [v2:172.21.15.151:6818/22315,v1:172.21.15.151:6819/22315] exists,up 6f7a9125-674c-4be2-93e1-b6296cd00f8e 2024-09-16T13:16:50.197 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 up in weight 1 up_from 14 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6800/22284,v1:172.21.15.151:6801/22284] [v2:172.21.15.151:6802/22284,v1:172.21.15.151:6803/22284] exists,up bcdfe435-f24d-45a9-a629-7380ad2e575a 2024-09-16T13:16:50.209 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6808/22271,v1:172.21.15.98:6809/22271] [v2:172.21.15.98:6810/22271,v1:172.21.15.98:6811/22271] exists,up be8543b4-3d28-4b37-a4ce-d2a56a09aeaf', 'osd.1 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6812/22272,v1:172.21.15.98:6815/22272] [v2:172.21.15.98:6818/22272,v1:172.21.15.98:6819/22272] exists,up a7f95c7b-56d4-437b-a160-cabb8fd2b1e4', 'osd.2 up in weight 1 up_from 14 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6800/22273,v1:172.21.15.98:6801/22273] [v2:172.21.15.98:6802/22273,v1:172.21.15.98:6803/22273] exists,up a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c', 'osd.3 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6824/22270,v1:172.21.15.98:6825/22270] [v2:172.21.15.98:6826/22270,v1:172.21.15.98:6827/22270] exists,up 4898b5b5-b912-430c-adcb-c52d90226917', 'osd.4 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6824/22286,v1:172.21.15.151:6825/22286] [v2:172.21.15.151:6826/22286,v1:172.21.15.151:6827/22286] exists,up 6c478be2-f606-41d8-967c-9a689f09a440', 'osd.5 up in weight 1 up_from 14 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6808/22285,v1:172.21.15.151:6809/22285] [v2:172.21.15.151:6810/22285,v1:172.21.15.151:6811/22285] exists,up 5001c828-b223-4d2c-b893-f9c56da73b83', 'osd.6 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6816/22315,v1:172.21.15.151:6817/22315] [v2:172.21.15.151:6818/22315,v1:172.21.15.151:6819/22315] exists,up 6f7a9125-674c-4be2-93e1-b6296cd00f8e', 'osd.7 up in weight 1 up_from 14 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6800/22284,v1:172.21.15.151:6801/22284] [v2:172.21.15.151:6802/22284,v1:172.21.15.151:6803/22284] exists,up bcdfe435-f24d-45a9-a629-7380ad2e575a'] 2024-09-16T13:16:50.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-16T13:16:50.305 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:50.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mon.a", 2024-09-16T13:16:50.305 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster": "ceph", 2024-09-16T13:16:50.305 INFO:teuthology.orchestra.run.smithi098.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T13:16:50.305 INFO:teuthology.orchestra.run.smithi098.stdout: "admin_socket_mode": "", 2024-09-16T13:16:50.305 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-16T13:16:50.305 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_client_required": "cephx, none", 2024-09-16T13:16:50.305 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_cluster_required": "cephx", 2024-09-16T13:16:50.305 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_debug": "true", 2024-09-16T13:16:50.305 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_service_required": "cephx", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_supported": "cephx", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_aio": "true", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_aio_reap_max": "16", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_async_discard": "false", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_block_size": "4096", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_debug_aio": "false", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T13:16:50.306 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_enable_discard": "false", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_flock_retry": "3", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_inject_crash": "0", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_ioring": "false", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_ioring_hipri": "false", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T13:16:50.307 INFO:teuthology.orchestra.run.smithi098.stdout: "bdev_type": "", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_allocator": "hybrid", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_buffered_io": "true", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T13:16:50.308 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_replay_recovery": "false", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_sync_write": "false", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_allocator": "hybrid", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_async_db_compaction": "true", 2024-09-16T13:16:50.309 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_create": "true", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_db_create": "false", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_db_path": "", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_db_size": "0", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_path": "", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T13:16:50.310 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_size": "107374182400", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_wal_create": "false", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_wal_path": "", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bluefs": "true", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T13:16:50.311 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_autotune": "true", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_size": "0", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_type": "2q", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T13:16:50.312 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_clone_cow": "true", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_mode": "none", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T13:16:50.313 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_freelist": "false", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T13:16:50.314 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T13:16:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fail_eio": "false", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T13:16:50.316 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_blob_size": "0", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T13:16:50.317 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-09-16T13:16:50.318 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_spdk_mem": "512", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T13:16:50.319 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_tracing": "false", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T13:16:50.320 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T13:16:50.321 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_cluster_require_version": "2", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_require_signatures": "false", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_require_version": "2", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_service_require_signatures": "false", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_service_require_version": "2", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "cephx_sign_messages": "true", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "chdir": "", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_acl_type": "", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_asio_thread_count": "2", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_cache_mid": "0.750000", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_cache_size": "16384", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_caps_release_delay": "5", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_check_pool_perm": "true", 2024-09-16T13:16:50.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_debug_force_sync_read": "false", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_debug_getattr_caps": "false", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_debug_inject_features": "", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_die_on_failed_remount": "false", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_dirsize_rbytes": "true", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_force_lazyio": "false", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_fs": "", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_inject_release_failure": "false", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_max_inline_size": "4096", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T13:16:50.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_mds_namespace": "", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_metadata": "", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_mount_gid": "-1", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_mount_timeout": "300", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_mount_uid": "-1", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_mountpoint": "/", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_notify_timeout": "10", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc": "true", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc_max_objects": "1000", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc_size": "209715200", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_permissions": "true", 2024-09-16T13:16:50.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_quota": "true", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_quota_df": "true", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_readahead_max_bytes": "0", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_readahead_max_periods": "4", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_readahead_min": "131072", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_reconnect_stale": "false", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_shutdown_timeout": "30", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_snapdir": ".snap", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_tick_interval": "1", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_trace": "", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_use_faked_inos": "false", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_use_random_mds": "false", 2024-09-16T13:16:50.325 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_graylog": "false", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_graylog_port": "12201", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_monitors": "default=true", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_syslog": "false", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "clog_to_syslog_level": "info", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "-", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_network": "", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_network_interface": "", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "compressor_zlib_isal": "false", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "compressor_zlib_level": "5", 2024-09-16T13:16:50.326 INFO:teuthology.orchestra.run.smithi098.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "compressor_zstd_level": "1", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location_hook": "", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location_hook_timeout": "10", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "daemonize": "false", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_alienstore": "0/5", 2024-09-16T13:16:50.327 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_asok": "1/5", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_asok_assert_abort": "false", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_auth": "20/20", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_bdev": "1/3", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_bluefs": "1/5", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_bluestore": "1/5", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_buffer": "0/1", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_cephsqlite": "0/5", 2024-09-16T13:16:50.328 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_civetweb": "1/10", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_client": "0/5", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_compressor": "1/5", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_context": "0/1", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_crush": "1/1", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_crypto": "1/5", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_dpdk": "1/5", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_eventtrace": "1/5", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_filer": "0/1", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_filestore": "1/3", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_finisher": "1/1", 2024-09-16T13:16:50.329 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_fuse": "1/5", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_javaclient": "1/5", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_journal": "1/3", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_journaler": "0/5", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_kstore": "1/5", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_leveldb": "4/5", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_lockdep": "0/1", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mclock": "1/5", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds": "1/5", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_balancer": "1/5", 2024-09-16T13:16:50.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_locker": "1/5", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_log": "1/5", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mds_migrator": "1/5", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_memdb": "4/5", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mgr": "2/5", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mgrc": "1/5", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_mon": "20/20", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_monc": "0/10", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_ms": "1/1", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_none": "0/5", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_objclass": "0/5", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_objectcacher": "0/5", 2024-09-16T13:16:50.331 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_objecter": "0/1", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_optracker": "0/5", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_osd": "1/5", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_paxos": "20/20", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_perfcounter": "1/5", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_prioritycache": "1/5", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rados": "0/5", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rbd": "0/5", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rbd_replay": "0/5", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_refs": "0/0", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_reserver": "1/1", 2024-09-16T13:16:50.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw": "1/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rgw_sync": "1/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_rocksdb": "4/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore": "0/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_cache": "0/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_device": "0/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_journal": "0/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_lba": "0/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_odata": "0/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_omap": "0/5", 2024-09-16T13:16:50.333 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_onode": "0/5", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_seastore_tm": "0/5", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_striper": "0/1", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_test": "0/5", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_throttle": "1/1", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_timer": "0/1", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "debug_tp": "0/5", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "device_failure_prediction_mode": "none", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "err_to_graylog": "false", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "err_to_journald": "false", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "err_to_stderr": "true", 2024-09-16T13:16:50.334 INFO:teuthology.orchestra.run.smithi098.stdout: "err_to_syslog": "false", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event_tracing": "false", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_http_port": "9926", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_prio_limit": "5", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_sort_metrics": "true", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "exporter_stats_period": "5", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "fake_statfs_for_testing": "0", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "fatal_signal_handlers": "true", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "filer_max_purge_ops": "10", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "filer_max_truncate_ops": "128", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_blackhole": "false", 2024-09-16T13:16:50.335 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_btrfs_snap": "true", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_caller_concurrency": "10", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_debug_omap_check": "false", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_debug_verify_split": "false", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_dump_file": "", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T13:16:50.336 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fadvise": "true", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fail_eio": "true", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fd_cache_size": "128", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fiemap": "false", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_inject_stall": "0", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_journal_parallel": "false", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_journal_trailing": "false", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_journal_writeahead": "false", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_kill_at": "0", 2024-09-16T13:16:50.337 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T13:16:50.338 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_merge_threshold": "-10", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_odsync_write": "false", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_omap_backend_path": "", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_op_threads": "2", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_punch_hole": "false", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T13:16:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_queue_max_ops": "50", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_seek_data_hole": "false", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_sloppy_crc": "false", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_splice": "false", 2024-09-16T13:16:50.340 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_split_multiple": "2", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_split_rand_factor": "20", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_update_to": "1000", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T13:16:50.341 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_xfs_extsize": "false", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "filestore_zfs_snap": "false", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "fio_dir": "/tmp/fio", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "6ddcae89-6c18-4eb7-8712-1c53f3b07278", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_allow_other": "true", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_big_writes": "true", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_debug": "false", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_default_permissions": "false", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_disable_pagecache": "false", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_max_write": "0", 2024-09-16T13:16:50.342 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_multithreaded": "true", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_require_active_mds": "true", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_set_user_groups": "true", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_splice_move": "true", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_splice_read": "true", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_splice_write": "true", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "gss_target_name": "ceph", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_file": "", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_inject_failure": "0", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_interval": "5", 2024-09-16T13:16:50.343 INFO:teuthology.orchestra.run.smithi098.stdout: "host": "smithi098", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T13:16:50.344 INFO:teuthology.orchestra.run.smithi098.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "inject_early_sigterm": "false", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "jaeger_tracing_enable": "false", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_aio": "true", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_align_min_size": "65536", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_block_align": "true", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_block_size": "4096", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_dio": "true", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_discard": "false", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_force_aio": "false", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_ignore_corruption": "false", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_max_write_entries": "100", 2024-09-16T13:16:50.345 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_replay_from": "0", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_write_header_frequency": "0", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "journal_zero_on_create": "false", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "journaler_prefetch_periods": "10", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "journaler_prezero_periods": "5", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "journaler_write_head_interval": "15", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "keyfile": "", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_backend": "rocksdb", 2024-09-16T13:16:50.346 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_max_bytes": "67108864", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_max_ops": "512", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_onode_map_size": "1024", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "kstore_sync_transaction": "false", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "leveldb_block_size": "65536", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "leveldb_bloom_size": "0", 2024-09-16T13:16:50.347 INFO:teuthology.orchestra.run.smithi098.stdout: "leveldb_cache_size": "536870912", 2024-09-16T13:16:50.348 INFO:teuthology.orchestra.run.smithi098.stdout: "leveldb_compact_on_mount": "false", 2024-09-16T13:16:50.348 INFO:teuthology.orchestra.run.smithi098.stdout: "leveldb_compression": "false", 2024-09-16T13:16:50.348 INFO:teuthology.orchestra.run.smithi098.stdout: "leveldb_log": "", 2024-09-16T13:16:50.348 INFO:teuthology.orchestra.run.smithi098.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-16T13:16:50.348 INFO:teuthology.orchestra.run.smithi098.stdout: "leveldb_max_open_files": "0", 2024-09-16T13:16:50.348 INFO:teuthology.orchestra.run.smithi098.stdout: "leveldb_paranoid": "false", 2024-09-16T13:16:50.348 INFO:teuthology.orchestra.run.smithi098.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-16T13:16:50.349 INFO:teuthology.orchestra.run.smithi098.stdout: "librados_thread_count": "2", 2024-09-16T13:16:50.349 INFO:teuthology.orchestra.run.smithi098.stdout: "lockdep": "false", 2024-09-16T13:16:50.349 INFO:teuthology.orchestra.run.smithi098.stdout: "lockdep_force_backtrace": "false", 2024-09-16T13:16:50.349 INFO:teuthology.orchestra.run.smithi098.stdout: "log_coarse_timestamps": "true", 2024-09-16T13:16:50.349 INFO:teuthology.orchestra.run.smithi098.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T13:16:50.349 INFO:teuthology.orchestra.run.smithi098.stdout: "log_flush_on_exit": "false", 2024-09-16T13:16:50.349 INFO:teuthology.orchestra.run.smithi098.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T13:16:50.349 INFO:teuthology.orchestra.run.smithi098.stdout: "log_graylog_port": "12201", 2024-09-16T13:16:50.349 INFO:teuthology.orchestra.run.smithi098.stdout: "log_max_new": "1000", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_max_recent": "10000", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_stderr_prefix": "", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": "true", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_graylog": "false", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_journald": "false", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_stderr": "false", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_syslog": "false", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_action_on_write_error": "1", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_alternate_name_max": "8192", 2024-09-16T13:16:50.350 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_asio_thread_count": "2", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_export_pin": "true", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_interval": "10", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_max": "-1", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_max_until": "-1", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_merge_size": "50", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T13:16:50.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_mode": "0", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_split_bits": "3", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_split_size": "10000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T13:16:50.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T13:16:50.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T13:16:50.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T13:16:50.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T13:16:50.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_mid": "0.700000", 2024-09-16T13:16:50.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T13:16:50.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T13:16:50.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T13:16:50.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_trim_interval": "1", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_debug_auth_pins": "false", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_debug_frag": "false", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_debug_scatterstat": "false", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_debug_subtrees": "false", 2024-09-16T13:16:50.354 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_default_dir_hash": "2", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_defer_session_stale": "true", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dir_max_entries": "0", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_early_reply": "true", 2024-09-16T13:16:50.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_enable_op_tracker": "true", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_enforce_unique_name": "true", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T13:16:50.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_health_dummy": "false", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_join_fs": "", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_journal_format": "1", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_create_at": "0", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_export_at": "0", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_import_at": "0", 2024-09-16T13:16:50.357 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_journal_at": "0", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_link_at": "0", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_openc_at": "0", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_rename_at": "0", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_max_events": "-1", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_max_segments": "128", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_pause": "false", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_segment_size": "0", 2024-09-16T13:16:50.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_completed_requests": "100000", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_export_size": "20971520", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_file_recover": "32", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_purge_files": "64", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_purge_ops": "8192", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_max_xattr_pairs_size": "65536", 2024-09-16T13:16:50.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_metrics_update_interval": "2", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_min_caps_per_client": "100", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_numa_node": "-1", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_history_duration": "600", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_history_size": "20", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_op_log_threshold": "5", 2024-09-16T13:16:50.360 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_ping_grace": "15", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_ping_interval": "5", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_max_caps": "30000", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_replay_interval": "1.000000", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_root_ino_gid": "0", 2024-09-16T13:16:50.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_root_ino_uid": "0", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_shutdown_check": "0", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_skip_ino": "0", 2024-09-16T13:16:50.362 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_snap_min_uid": "0", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_snap_rstat": "false", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_symlink_recovery": "true", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_thrash_exports": "0", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_thrash_fragments": "0", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_tick_interval": "5.000000", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_valgrind_exit": "false", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_verify_backtrace": "1", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_verify_scatter": "false", 2024-09-16T13:16:50.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T13:16:50.364 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_wipe_sessions": "false", 2024-09-16T13:16:50.364 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool_debug": "false", 2024-09-16T13:16:50.364 INFO:teuthology.orchestra.run.smithi098.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T13:16:50.364 INFO:teuthology.orchestra.run.smithi098.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T13:16:50.364 INFO:teuthology.orchestra.run.smithi098.stdout: "memstore_page_set": "false", 2024-09-16T13:16:50.364 INFO:teuthology.orchestra.run.smithi098.stdout: "memstore_page_size": "65536", 2024-09-16T13:16:50.364 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_client_bytes": "134217728", 2024-09-16T13:16:50.364 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_client_messages": "512", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_mds_messages": "128", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T13:16:50.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_mon_messages": "128", 2024-09-16T13:16:50.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T13:16:50.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_osd_messages": "8192", 2024-09-16T13:16:50.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_pool": "true", 2024-09-16T13:16:50.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T13:16:50.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_standby_modules": "true", 2024-09-16T13:16:50.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_stats_period": "5", 2024-09-16T13:16:50.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_stats_threshold": "5", 2024-09-16T13:16:50.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_tick_period": "2", 2024-09-16T13:16:50.366 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_allow_pool_delete": "true", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_allow_pool_size_one": "true", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_bytes": "104857600", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_directed_command_retry": "5", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T13:16:50.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T13:16:50.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T13:16:50.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T13:16:50.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T13:16:50.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T13:16:50.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T13:16:50.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T13:16:50.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T13:16:50.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T13:16:50.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T13:16:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_level": "debug", 2024-09-16T13:16:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T13:16:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T13:16:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T13:16:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T13:16:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_journald": "false", 2024-09-16T13:16:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T13:16:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T13:16:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T13:16:50.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T13:16:50.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_compact_on_start": "false", 2024-09-16T13:16:50.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_compact_on_trim": "true", 2024-09-16T13:16:50.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T13:16:50.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T13:16:50.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T13:16:50.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_cpu_threads": "4", 2024-09-16T13:16:50.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T13:16:50.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T13:16:50.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T13:16:50.370 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_data_avail_crit": "5", 2024-09-16T13:16:50.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_data_avail_warn": "5", 2024-09-16T13:16:50.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T13:16:50.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T13:16:50.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T13:16:50.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_dump_json": "false", 2024-09-16T13:16:50.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T13:16:50.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T13:16:50.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_extra_checks": "false", 2024-09-16T13:16:50.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T13:16:50.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T13:16:50.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_no_require_pacific": "false", 2024-09-16T13:16:50.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_no_require_quincy": "false", 2024-09-16T13:16:50.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T13:16:50.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T13:16:50.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T13:16:50.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_down_mkfs_grace": "120", 2024-09-16T13:16:50.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_election_default_strategy": "1", 2024-09-16T13:16:50.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_election_timeout": "5.000000", 2024-09-16T13:16:50.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T13:16:50.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T13:16:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T13:16:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_enable_op_tracker": "true", 2024-09-16T13:16:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_fake_pool_delete": "false", 2024-09-16T13:16:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_force_quorum_join": "false", 2024-09-16T13:16:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T13:16:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_detail_to_clog": "false", 2024-09-16T13:16:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_log_update_period": "5", 2024-09-16T13:16:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_max_detail": "50", 2024-09-16T13:16:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_to_clog": "true", 2024-09-16T13:16:50.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T13:16:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T13:16:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_host": "172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790],172.21.15.151", 2024-09-16T13:16:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_host_override": "", 2024-09-16T13:16:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_initial_members": "", 2024-09-16T13:16:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T13:16:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T13:16:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T13:16:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T13:16:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T13:16:50.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_lease": "5.000000", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_log_full_interval": "50", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_log_max": "10000", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_log_max_summary": "50", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_log_epochs": "500", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_osd": "10000", 2024-09-16T13:16:50.375 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T13:16:50.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T13:16:50.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T13:16:50.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T13:16:50.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T13:16:50.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T13:16:50.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_memory_autotune": "true", 2024-09-16T13:16:50.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_memory_target": "2147483648", 2024-09-16T13:16:50.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T13:16:50.376 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_digest_period": "5", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_min_osdmap_epochs": "500", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_complaint_time": "30", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_history_duration": "600", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_history_size": "20", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_op_log_threshold": "5", 2024-09-16T13:16:50.377 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_cache_size": "500", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T13:16:50.378 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T13:16:50.379 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_report_timeout": "900", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T13:16:50.380 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_reweight_max_osds": "4", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T13:16:50.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_scrub_interval": "86400", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_scrub_max_keys": "100", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_scrub_timeout": "300", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_session_timeout": "300", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_smart_report_timeout": "5", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_stretch_pool_size": "4", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_debug": "false", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T13:16:50.382 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_tick_interval": "5", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T13:16:50.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_older_version": "true", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T13:16:50.384 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": "", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_op_threads": "3", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_cm": "false", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_device_name": "", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T13:16:50.385 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_sl": "3", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_rdma_type": "ib", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_async_reap_threshold": "5", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_before_connect": "false", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_ipv4": "true", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_ipv6": "false", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_msgr1": "true", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_msgr2": "true", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_port_max": "7568", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_port_min": "6800", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T13:16:50.386 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_retry_count": "3", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_bind_retry_delay": "5", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_blackhole_client": "false", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_blackhole_mds": "false", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_blackhole_mgr": "false", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_blackhole_mon": "false", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_blackhole_osd": "false", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_client_mode": "crc secure", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_cluster_type": "", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_compress_secure": "false", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_crc_data": "true", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_crc_header": "true", 2024-09-16T13:16:50.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_die_on_bug": "true", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_die_on_old_message": "true", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_hugepages": "", 2024-09-16T13:16:50.388 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_lro": "true", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_pmd": "", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_port_id": "0", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_dump_on_send": "false", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T13:16:50.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_delay_type": "", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_inject_socket_failures": "5000", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_max_accept_failures": "4", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_max_backoff": "15.000000", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_osd_compress_mode": "none", 2024-09-16T13:16:50.390 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_pq_min_cost": "65536", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_public_type": "", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_service_mode": "crc secure", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_tcp_nodelay": "true", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "ms_type": "async+posix", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "no_config_file": "false", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T13:16:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_inflight_ops": "1024", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter_timeout": "10.000000", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "objectstore_blackhole": "false", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "openssl_engine_opts": "", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_max_ops": "4", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T13:16:50.392 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_agent_slop": "0.020000", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_asio_thread_count": "2", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_async_recovery_min_cost": "100", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backfill_scan_max": "512", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backfill_scan_min": "64", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backoff_on_peering": "false", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_beacon_report_interval": "300", 2024-09-16T13:16:50.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bench_duration": "30", 2024-09-16T13:16:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T13:16:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T13:16:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T13:16:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_blkin_trace_all": "false", 2024-09-16T13:16:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T13:16:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T13:16:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T13:16:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T13:16:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T13:16:50.394 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-16T13:16:50.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T13:16:50.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_class_update_on_start": "true", 2024-09-16T13:16:50.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_client_message_cap": "256", 2024-09-16T13:16:50.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T13:16:50.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_client_op_priority": "63", 2024-09-16T13:16:50.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_client_watch_timeout": "30", 2024-09-16T13:16:50.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_command_max_records": "256", 2024-09-16T13:16:50.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_compact_on_start": "false", 2024-09-16T13:16:50.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T13:16:50.395 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_crush_update_on_start": "true", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T13:16:50.396 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T13:16:50.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-16T13:16:50.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T13:16:50.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T13:16:50.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T13:16:50.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_op_order": "false", 2024-09-16T13:16:50.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T13:16:50.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T13:16:50.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T13:16:50.397 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T13:16:50.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_shutdown": "false", 2024-09-16T13:16:50.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T13:16:50.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T13:16:50.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T13:16:50.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T13:16:50.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T13:16:50.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T13:16:50.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T13:16:50.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T13:16:50.398 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_default_notify_timeout": "30", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T13:16:50.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_enable_op_tracker": "true", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_fast_info": "true", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_fast_shutdown": "true", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_function_tracing": "false", 2024-09-16T13:16:50.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_grace": "20", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_interval": "6", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_stale": "600", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T13:16:50.401 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T13:16:50.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T13:16:50.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T13:16:50.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T13:16:50.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_journal_size": "5120", 2024-09-16T13:16:50.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_kill_backfill_at": "0", 2024-09-16T13:16:50.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T13:16:50.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_map_cache_size": "50", 2024-09-16T13:16:50.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_map_dedup": "true", 2024-09-16T13:16:50.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_map_message_max": "40", 2024-09-16T13:16:50.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T13:16:50.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T13:16:50.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_attr_name_len": "100", 2024-09-16T13:16:50.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_attr_size": "0", 2024-09-16T13:16:50.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_backfills": "1", 2024-09-16T13:16:50.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_markdown_count": "5", 2024-09-16T13:16:50.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_markdown_period": "600", 2024-09-16T13:16:50.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_object_name_len": "2048", 2024-09-16T13:16:50.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T13:16:50.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_object_size": "134217728", 2024-09-16T13:16:50.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T13:16:50.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T13:16:50.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T13:16:50.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_pg_log_entries": "10000", 2024-09-16T13:16:50.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T13:16:50.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_pgls": "1024", 2024-09-16T13:16:50.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_push_cost": "8388608", 2024-09-16T13:16:50.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_push_objects": "10", 2024-09-16T13:16:50.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_scrubs": "3", 2024-09-16T13:16:50.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T13:16:50.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T13:16:50.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T13:16:50.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_max_write_size": "90", 2024-09-16T13:16:50.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T13:16:50.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T13:16:50.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T13:16:50.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T13:16:50.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T13:16:50.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T13:16:50.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T13:16:50.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T13:16:50.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_profile": "balanced", 2024-09-16T13:16:50.406 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T13:16:50.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T13:16:50.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T13:16:50.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T13:16:50.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T13:16:50.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T13:16:50.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T13:16:50.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T13:16:50.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T13:16:50.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T13:16:50.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T13:16:50.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_base": "805306368", 2024-09-16T13:16:50.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T13:16:50.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T13:16:50.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T13:16:50.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_target": "4294967296", 2024-09-16T13:16:50.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_target_autotune": "false", 2024-09-16T13:16:50.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T13:16:50.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_min_pg_log_entries": "250", 2024-09-16T13:16:50.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_min_recovery_priority": "0", 2024-09-16T13:16:50.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T13:16:50.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T13:16:50.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mon_report_interval": "5", 2024-09-16T13:16:50.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T13:16:50.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T13:16:50.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_num_cache_shards": "32", 2024-09-16T13:16:50.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T13:16:50.409 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_numa_node": "-1", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_objecter_finishers": "1", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_objectstore": "bluestore", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_objectstore_fuse": "false", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_objectstore_tracing": "false", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_history_duration": "600", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_history_size": "20", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T13:16:50.410 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_log_threshold": "5", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_shards": "0", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_op_thread_timeout": "15", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_open_classes_on_start": "true", 2024-09-16T13:16:50.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_os_flags": "0", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_peering_op_priority": "255", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_delete_priority": "5", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_log_trim_min": "100", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T13:16:50.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T13:16:50.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T13:16:50.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T13:16:50.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T13:16:50.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T13:16:50.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T13:16:50.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T13:16:50.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T13:16:50.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T13:16:50.413 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_flags": "0", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_min_size": "0", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_size": "2", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_default_type": "replicated", 2024-09-16T13:16:50.414 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_cost": "20971520", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_active": "0", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T13:16:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_op_priority": "3", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_priority": "5", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_repair_during_recovery": "false", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T13:16:50.416 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_chunk_max": "25", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_cost": "52428800", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_end_hour": "0", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T13:16:50.417 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_priority": "5", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_skip_data_digest": "false", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_smart_report_timeout": "5", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_priority": "5", 2024-09-16T13:16:50.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_target_transaction_size": "30", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T13:16:50.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_tracing": "false", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_use_stale_snap": "false", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_kill_at": "0", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_max_join_drift": "10", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_min": "500", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_min_wait": "0.050000", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_service_trim_max": "500", 2024-09-16T13:16:50.420 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_service_trim_min": "250", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_stash_full_interval": "25", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_trim_max": "500", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "paxos_trim_min": "250", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "perf": "true", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "-", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrv": "", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_bind_addr": "-", 2024-09-16T13:16:50.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_network": "", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "public_network_interface": "", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "qat_compressor_enabled": "false", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rados_mon_op_timeout": "0", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rados_osd_op_timeout": "0", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rados_tracing": "false", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_atime_update_interval": "60", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache": "true", 2024-09-16T13:16:50.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_policy": "writearound", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_size": "33554432", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_compression_hint": "none", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_clone_format": "auto", 2024-09-16T13:16:50.423 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_data_pool": "", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_features": "61", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_format": "2", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_map_options": "", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_order": "22", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_pool": "rbd", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_stripe_count": "0", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T13:16:50.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_io_scheduler": "simple", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_order": "24", 2024-09-16T13:16:50.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_pool": "", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_journal_splay_width": "4", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T13:16:50.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T13:16:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T13:16:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T13:16:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T13:16:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T13:16:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T13:16:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T13:16:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T13:16:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T13:16:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T13:16:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T13:16:50.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T13:16:50.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T13:16:50.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T13:16:50.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T13:16:50.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T13:16:50.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T13:16:50.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T13:16:50.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T13:16:50.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_op_threads": "1", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T13:16:50.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_plugins": "", 2024-09-16T13:16:50.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T13:16:50.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T13:16:50.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T13:16:50.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T13:16:50.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T13:16:50.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T13:16:50.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T13:16:50.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T13:16:50.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T13:16:50.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T13:16:50.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T13:16:50.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T13:16:50.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T13:16:50.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T13:16:50.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T13:16:50.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T13:16:50.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T13:16:50.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T13:16:50.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_tracing": "false", 2024-09-16T13:16:50.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_validate_names": "true", 2024-09-16T13:16:50.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_validate_pool": "true", 2024-09-16T13:16:50.433 INFO:teuthology.orchestra.run.smithi098.stdout: "restapi_base_url": "", 2024-09-16T13:16:50.433 INFO:teuthology.orchestra.run.smithi098.stdout: "restapi_log_level": "", 2024-09-16T13:16:50.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T13:16:50.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_admin_entry": "admin", 2024-09-16T13:16:50.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_backend_store": "rados", 2024-09-16T13:16:50.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_barbican_url": "", 2024-09-16T13:16:50.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_beast_enable_async": "true", 2024-09-16T13:16:50.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T13:16:50.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T13:16:50.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T13:16:50.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T13:16:50.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T13:16:50.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_cache_enabled": "true", 2024-09-16T13:16:50.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T13:16:50.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_cache_lru_size": "10000", 2024-09-16T13:16:50.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_content_length_compat": "false", 2024-09-16T13:16:50.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T13:16:50.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T13:16:50.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T13:16:50.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_cross_domain_policy": "", 2024-09-16T13:16:50.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T13:16:50.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T13:16:50.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T13:16:50.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T13:16:50.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T13:16:50.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T13:16:50.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T13:16:50.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T13:16:50.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T13:16:50.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T13:16:50.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T13:16:50.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T13:16:50.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T13:16:50.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T13:16:50.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T13:16:50.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T13:16:50.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T13:16:50.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T13:16:50.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T13:16:50.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T13:16:50.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T13:16:50.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T13:16:50.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T13:16:50.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T13:16:50.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T13:16:50.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T13:16:50.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T13:16:50.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T13:16:50.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T13:16:50.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T13:16:50.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T13:16:50.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T13:16:50.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T13:16:50.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T13:16:50.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T13:16:50.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T13:16:50.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T13:16:50.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T13:16:50.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T13:16:50.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T13:16:50.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T13:16:50.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_log_window": "30", 2024-09-16T13:16:50.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_data_notify_interval_msec": "200", 2024-09-16T13:16:50.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T13:16:50.440 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T13:16:50.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dns_name": "", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dns_s3website_name": "", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T13:16:50.442 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_ops_log": "false", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_static_website": "false", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enable_usage_log": "false", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_expose_bucket": "false", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_extended_http_attrs": "", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T13:16:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_objs": "32", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_init_timeout": "300", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T13:16:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_password": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_project": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_token": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_admin_user": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_api_version": "2", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T13:16:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_url": "", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_max_objs": "32", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_max_worker": "3", 2024-09-16T13:16:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ldap_uri": "", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_log_http_headers": "", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T13:16:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_attr_size": "0", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_listing_results": "1000", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_notify_retries": "10", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T13:16:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T13:16:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T13:16:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T13:16:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T13:16:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T13:16:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T13:16:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T13:16:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T13:16:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T13:16:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_num_control_oids": "8", 2024-09-16T13:16:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_numa_node": "-1", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_op_tracing": "false", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_opa_token": "", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_opa_url": "", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T13:16:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ops_log_file_path": "", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ops_log_rados": "true", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_print_continue": "true", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T13:16:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_rados_tracing": "false", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_realm": "", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_realm_id": "", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_region": "", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T13:16:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_request_uri": "", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_resolve_cname": "false", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_run_sync_thread": "true", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T13:16:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_script_uri": "", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_service_provider_name": "", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_client_id": "", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_client_secret": "", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_entry": "sts", 2024-09-16T13:16:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_key": "sts", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_auth_url": "", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_custom_header": "", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_need_stats": "true", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_tenant_name": "", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T13:16:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_url": "", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_lease_period": "120", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T13:16:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_thread_pool_size": "512", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_comment": "", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_createby": "", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_encoding": "", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_flag": "false", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_origin": "", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_torrent_tracker": "", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_usage_max_shards": "32", 2024-09-16T13:16:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_use_opa_authz": "false", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_user_unique_email": "true", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_verify_ssl": "true", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zone": "", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zone_id": "", 2024-09-16T13:16:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zonegroup": "", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zonegroup_id": "", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_block_size": "4096", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T13:16:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_partition_filters": "false", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_perf": "false", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T13:16:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "run_dir": "/var/run/ceph", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_block_create": "true", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_device_size": "107374182400", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "seastore_segment_size": "67108864", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "setgroup": "", 2024-09-16T13:16:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "setuser": "", 2024-09-16T13:16:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "setuser_match_path": "", 2024-09-16T13:16:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T13:16:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "thp": "false", 2024-09-16T13:16:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "threadpool_default_timeout": "60", 2024-09-16T13:16:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T13:16:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "throttler_perf_counter": "true" 2024-09-16T13:16:50.463 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:50.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T13:16:50.606 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a: {} 2024-09-16T13:16:50.619 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b: {} 2024-09-16T13:16:50.633 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c: {} 2024-09-16T13:16:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:16:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:16:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:16:50.649 DEBUG:teuthology.orchestra.run.smithi098:> type ceph-objectstore-tool 2024-09-16T13:16:50.655 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T13:16:50.655 DEBUG:teuthology.orchestra.run.smithi151:> type ceph-objectstore-tool 2024-09-16T13:16:50.661 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T13:16:50.662 INFO:teuthology.run_tasks:Running task rbd_fsx... 2024-09-16T13:16:50.673 INFO:tasks.rbd_fsx:starting rbd_fsx... 2024-09-16T13:16:50.674 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T13:16:50.674 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:16:50.674 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:16:50.674 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-16T13:16:50.674 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:16:50.674 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:16:50.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-16T13:16:50.675 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T13:16:50.675 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T13:16:50.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:16:50.676 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T13:16:50.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:16:50.676 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T13:16:50.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:16:50.677 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 6000 pool_client.0 image_client.0 2024-09-16T13:16:50.678 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:50.668+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:16:50.706 INFO:teuthology.orchestra.run.smithi098.stdout:mapped writes DISABLED 2024-09-16T13:16:50.706 INFO:teuthology.orchestra.run.smithi098.stdout:mapped reads DISABLED 2024-09-16T13:16:50.706 INFO:teuthology.orchestra.run.smithi098.stdout:Seed set to 2013 2024-09-16T13:16:50.777 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:50.772+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:50.810 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:16:50.812 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:50.812 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T13:16:50.812 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:50.826 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:16:50.831 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:50.831 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:50.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:16:50.849 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:16:50.867 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:16:50.878 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:50.871+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:16:50.887 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:16:50.910 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:16:50.933 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:50.933 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:50.937 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:16:50.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:16:50.961 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:16:50.963 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:16:50.963 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:16:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:16:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:16:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:16:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:16:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:16:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:16:50.978 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:50.971+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:16:51.038 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:51.038 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:51.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:16:51.079 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:51.072+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:51.145 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:51.145 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:51.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:16:51.197 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:51.192+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:51.282 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:51.276+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:51.294 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:16:51.306 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:51.307 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:51.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:16:51.383 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:51.376+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:16:51.410 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:51.410 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:51.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:16:51.484 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:51.476+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:51.520 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:51.520 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:51.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:16:51.584 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:51.579+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:16:51.676 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:51.676 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:51.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:16:51.686 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:51.680+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:51.783 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:51.783 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:51.786 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:51.780+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:16:51.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:16:51.887 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:51.882+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:16:51.938 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:51.938 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:51.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:16:51.987 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:51.980+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:16:52.040 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:52.040 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:52.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:16:52.089 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:52.082+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:52.189 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:52.178+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:16:52.196 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:52.196 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:52.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:16:52.295 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:52.282+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:52.321 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:52.321 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:52.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:16:52.392 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:52.382+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:52.422 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:52.422 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:52.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:16:52.493 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:52.482+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:16:52.529 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:52.529 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:52.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:16:52.593 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:52.588+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:16:52.684 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:52.684 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:52.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:16:52.694 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:52.688+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:52.790 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:52.790 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:52.794 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:52.788+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:52.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:16:52.895 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:52.888+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:52.900 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:52.900 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:52.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:16:52.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:16:52.997 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:52.986+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:16:53.010 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:53.010 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:53.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:16:53.097 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:53.086+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:53.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:16:53.148 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:16:53.169 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:53.169 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:53.170 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:16:53.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:16:53.189 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:16:53.198 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:53.190+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:16:53.208 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:16:53.231 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:16:53.253 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:16:53.275 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:16:53.277 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:16:53.278 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:16:53.278 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:16:53.278 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:16:53.278 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:16:53.278 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:16:53.278 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:16:53.278 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:16:53.279 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:16:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:16:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [] 2024-09-16T13:16:53.279 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:53.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:16:53.299 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:53.292+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:16:53.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:16:53.400 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:53.390+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:16:53.432 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:53.432 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:53.432 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:16:53.432 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:53.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:16:53.500 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:53.490+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:53.543 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:53.543 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:53.543 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:16:53.543 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:16:53.543 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:53.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:16:53.602 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:53.596+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:16:53.649 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.024918+0000", 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.616624893000001, 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0098686339999999994, 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 44 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024918+0000", 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024918+0000", 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024919+0000", 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.093e-06 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024997+0000", 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8595000000000002e-05 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025009+0000", 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2006e-05 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025016+0000", 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999999e-06 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.031767+0000", 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067515229999999997 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.031892+0000", 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012455100000000001 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.032088+0000", 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000196152 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034632+0000", 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025435430000000001 2024-09-16T13:16:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034658+0000", 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6784999999999999e-05 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034734+0000", 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5482000000000007e-05 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034786+0000", 2024-09-16T13:16:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2173999999999999e-05 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025030+0000", 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.616512306000001, 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011426475, 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025030+0000", 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025030+0000", 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025031+0000", 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0100000000000001e-06 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025037+0000", 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4600000000000002e-06 2024-09-16T13:16:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025046+0000", 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2259999999999994e-06 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025053+0000", 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0140000000000001e-06 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033067+0000", 2024-09-16T13:16:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080137490000000006 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033146+0000", 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9068999999999998e-05 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033263+0000", 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117516 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034798+0000", 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001535177 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036385+0000", 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015866509999999999 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036425+0000", 2024-09-16T13:16:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0417999999999998e-05 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036457+0000", 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1185e-05 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:16:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.616482785999999, 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013323559, 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 46 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:16:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025065+0000", 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7299999999999996e-06 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025076+0000", 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0943e-05 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025081+0000", 2024-09-16T13:16:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.48e-06 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033698+0000", 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086164589999999999 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033803+0000", 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010459300000000001 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033994+0000", 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000191022 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037370+0000", 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033765589999999999 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038307+0000", 2024-09-16T13:16:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093719199999999999 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038346+0000", 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9033999999999999e-05 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038383+0000", 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6831e-05 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.616452871, 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014352281999999999, 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 47 2024-09-16T13:16:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0699999999999996e-07 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025220+0000", 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012995699999999999 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025231+0000", 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1297e-05 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025238+0000", 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.263e-06 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034888+0000", 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0096507490000000001 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035006+0000", 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117416 2024-09-16T13:16:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035143+0000", 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013723699999999999 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037388+0000", 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022444710000000001 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039332+0000", 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019441930000000001 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039397+0000", 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5289000000000005e-05 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039442+0000", 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4703000000000002e-05 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025246+0000", 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.616296404, 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014350958, 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 48 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025246+0000", 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025246+0000", 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025247+0000", 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7000000000000003e-07 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025252+0000", 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6670000000000004e-06 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025260+0000", 2024-09-16T13:16:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6989999999999992e-06 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025265+0000", 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0300000000000001e-06 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035652+0000", 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010386687 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035734+0000", 2024-09-16T13:16:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2108999999999996e-05 2024-09-16T13:16:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035898+0000", 2024-09-16T13:16:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016392499999999999 2024-09-16T13:16:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039267+0000", 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033692409999999998 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039511+0000", 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024390499999999999 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039561+0000", 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9339000000000003e-05 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039597+0000", 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6386e-05 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025294+0000", 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.616248081999998, 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015060367999999999, 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 49 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025294+0000", 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025294+0000", 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025295+0000", 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025301+0000", 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5899999999999998e-06 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025356+0000", 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5495000000000003e-05 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025366+0000", 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0047000000000001e-05 2024-09-16T13:16:53.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036507+0000", 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011140768000000001 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036635+0000", 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128458 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036907+0000", 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0002719 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039665+0000", 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027574269999999998 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040255+0000", 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00058990500000000005 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040314+0000", 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9301e-05 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040355+0000", 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0717000000000003e-05 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.687 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.616156830000001, 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:16:53.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.616009424000001, 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:16:53.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:16:53.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.615982080999999, 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.700 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:16:53.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.705 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583774+0000", 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.057768421999999, 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011465823999999999, 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 69 2024-09-16T13:16:53.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583774+0000", 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583774+0000", 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583775+0000", 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583847+0000", 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2051999999999994e-05 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583903+0000", 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6165000000000002e-05 2024-09-16T13:16:53.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583923+0000", 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9979999999999998e-05 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.589618+0000", 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056950400000000002 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.589858+0000", 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000239764 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.590239+0000", 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038074000000000002 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595048+0000", 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004809603 2024-09-16T13:16:53.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595130+0000", 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1903999999999994e-05 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595195+0000", 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4491000000000003e-05 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595240+0000", 2024-09-16T13:16:53.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5034999999999998e-05 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583939+0000", 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.057603541999999, 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013326655999999999, 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 70 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583939+0000", 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583939+0000", 2024-09-16T13:16:53.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583940+0000", 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6300000000000004e-07 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583945+0000", 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583956+0000", 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1147e-05 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583962+0000", 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5559999999999998e-06 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.592841+0000", 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0088788419999999996 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.592926+0000", 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5424000000000001e-05 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.593071+0000", 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145093 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596234+0000", 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031626990000000002 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597165+0000", 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093124799999999997 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597219+0000", 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3680000000000001e-05 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597266+0000", 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6801000000000001e-05 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583971+0000", 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.057571413000002, 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01344849, 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.718 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 71 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583971+0000", 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583971+0000", 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583972+0000", 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1399999999999996e-07 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583978+0000", 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.37e-06 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583987+0000", 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7360000000000007e-06 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583992+0000", 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1229999999999999e-06 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.593635+0000", 2024-09-16T13:16:53.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0096431659999999999 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.593847+0000", 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021216400000000001 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.594195+0000", 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034803999999999998 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597283+0000", 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030879169999999999 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597330+0000", 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6465999999999999e-05 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597380+0000", 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0698999999999997e-05 2024-09-16T13:16:53.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597419+0000", 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9094999999999998e-05 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.057543720000002, 2024-09-16T13:16:53.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015940290999999999, 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 72 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4600000000000004e-07 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584119+0000", 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120004 2024-09-16T13:16:53.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584145+0000", 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5539999999999999e-05 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584155+0000", 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9729999999999992e-06 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595284+0000", 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011129139999999999 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595404+0000", 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120325 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595569+0000", 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000164556 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597886+0000", 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023171490000000001 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599784+0000", 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001898278 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599899+0000", 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114973 2024-09-16T13:16:53.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599939+0000", 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9607000000000002e-05 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584171+0000", 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.057371500999999, 2024-09-16T13:16:53.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016071118999999998, 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 73 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584171+0000", 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584171+0000", 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584172+0000", 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.24e-06 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584183+0000", 2024-09-16T13:16:53.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0393e-05 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584210+0000", 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7679999999999999e-05 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584221+0000", 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0583e-05 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596305+0000", 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012084068999999999 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596419+0000", 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011364799999999999 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596560+0000", 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014121000000000001 2024-09-16T13:16:53.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599720+0000", 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031602819999999999 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600159+0000", 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000438818 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600201+0000", 2024-09-16T13:16:53.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2342999999999998e-05 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600242+0000", 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0852999999999997e-05 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:16:53.736 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.057306819000001, 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.742 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.057219227000001, 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:16:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:16:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:16:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:16:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:16:53.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.057049305, 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.749 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:16:53.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:16:53.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:92 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:36.382227+0000", 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.259314906, 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011280221999999999, 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.755 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382227+0000", 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382227+0000", 2024-09-16T13:16:53.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382228+0000", 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9999999999999996e-07 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382243+0000", 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4286e-05 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382251+0000", 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5329999999999994e-06 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382259+0000", 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8639999999999996e-06 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.388758+0000", 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064991240000000002 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.388838+0000", 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9987000000000005e-05 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.388964+0000", 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012593000000000001 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.392626+0000", 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036619159999999999 2024-09-16T13:16:53.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393309+0000", 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068294700000000005 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393432+0000", 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123387 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393508+0000", 2024-09-16T13:16:53.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5448e-05 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:93 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.259271724000001, 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011390030000000001, 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.761 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 93 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:16:53.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8400000000000003e-07 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382384+0000", 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0001128 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382396+0000", 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1987000000000001e-05 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382403+0000", 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6760000000000001e-06 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.389370+0000", 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069670080000000002 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.389449+0000", 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9448000000000002e-05 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.389563+0000", 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011318 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.392716+0000", 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031536350000000001 2024-09-16T13:16:53.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393522+0000", 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080626500000000002 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393602+0000", 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9528000000000002e-05 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393661+0000", 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8718999999999998e-05 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.259130887000001, 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013820321999999999, 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:53.767 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 94 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:16:53.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2699999999999998e-07 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382418+0000", 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1390000000000003e-06 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382426+0000", 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6770000000000001e-06 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382435+0000", 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1330000000000005e-06 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.391103+0000", 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086678729999999996 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.391194+0000", 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0740000000000002e-05 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.391327+0000", 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013291 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.394978+0000", 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003651502 2024-09-16T13:16:53.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.396103+0000", 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011249439999999999 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.396180+0000", 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6783e-05 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:53.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.396232+0000", 2024-09-16T13:16:53.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1693999999999997e-05 2024-09-16T13:16:53.773 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.773 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.773 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.773 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:53.773 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:53.773 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:53.774 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:53.766+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:16:53.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:16:53.875 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:53.870+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:16:53.882 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:16:53.882 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:16:53.882 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:53.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:16:53.976 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:53.970+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:16:53.991 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:53.991 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:16:53.991 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:16:53.991 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:16:53.991 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:54.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:16:54.076 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:54.072+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:16:54.105 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:54.105 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:16:54.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:16:54.105 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:16:54.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.105 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:51 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'33, mlcod=16'27)", 2024-09-16T13:16:54.105 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:16:54.105 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.059518579999999, 2024-09-16T13:16:54.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065364949999999998, 2024-09-16T13:16:54.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038562+0000", 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0920000000000001e-06 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038571+0000", 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1339999999999993e-06 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038600+0000", 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9254000000000001e-05 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038609+0000", 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6049999999999998e-06 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038982+0000", 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037359699999999998 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039013+0000", 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0945000000000003e-05 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045052+0000", 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060389490000000001 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045097+0000", 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4919000000000001e-05 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.109 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:52 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'34, mlcod=16'29)", 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.039817+0000", 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.058262733999999, 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057720159999999996, 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039817+0000", 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039817+0000", 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039818+0000", 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.223e-06 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039828+0000", 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0125e-05 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039851+0000", 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3485999999999998e-05 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039861+0000", 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909999999999996e-06 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039933+0000", 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1829999999999995e-05 2024-09-16T13:16:54.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039967+0000", 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4375999999999997e-05 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045505+0000", 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055381470000000002 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045589+0000", 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3537999999999999e-05 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:54 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.047292+0000", 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.050786923, 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042292679999999996, 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.114 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047292+0000", 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047292+0000", 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047293+0000", 2024-09-16T13:16:54.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047444+0000", 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000151345 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047455+0000", 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1011e-05 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047463+0000", 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3159999999999999e-06 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047531+0000", 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8484000000000004e-05 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047604+0000", 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3034999999999994e-05 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.051446+0000", 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038415739999999999 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.051522+0000", 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5891999999999998e-05 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:55 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-16T13:16:54.118 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.049946796, 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062294560000000004, 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:16:54.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5000000000000002e-07 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048240+0000", 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107209 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048252+0000", 2024-09-16T13:16:54.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1263999999999999e-05 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048263+0000", 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1685000000000001e-05 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048355+0000", 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2033999999999993e-05 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048378+0000", 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3062e-05 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054305+0000", 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059262639999999997 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054362+0000", 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7287999999999998e-05 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:56 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'38, mlcod=16'33)", 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.049230777999998, 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055458679999999998, 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0800000000000004e-07 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048942+0000", 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2954000000000005e-05 2024-09-16T13:16:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048954+0000", 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2234e-05 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048961+0000", 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0530000000000003e-06 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.049066+0000", 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105079 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.049088+0000", 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2427999999999999e-05 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054366+0000", 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052782189999999998 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054394+0000", 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8192999999999999e-05 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:65 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'43, mlcod=16'40)", 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.584665+0000", 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.513414241, 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003952952, 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584665+0000", 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584665+0000", 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584667+0000", 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.717e-06 2024-09-16T13:16:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584679+0000", 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2533e-05 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584710+0000", 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0648999999999997e-05 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584721+0000", 2024-09-16T13:16:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.114e-05 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584832+0000", 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110697 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584914+0000", 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2504999999999997e-05 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.588521+0000", 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036070759999999999 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.588618+0000", 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6634999999999999e-05 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:66 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'40)", 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.585658+0000", 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.512421412999998, 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004136504, 2024-09-16T13:16:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585658+0000", 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585658+0000", 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585660+0000", 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.903e-06 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585780+0000", 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000120041 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585847+0000", 2024-09-16T13:16:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7504999999999998e-05 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585863+0000", 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5546e-05 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585961+0000", 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7794999999999995e-05 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.586018+0000", 2024-09-16T13:16:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6888999999999997e-05 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.589719+0000", 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037009669999999999 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.589794+0000", 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5858000000000005e-05 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:72 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'50, mlcod=16'46)", 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.595438+0000", 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.502641171, 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003137623, 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.595438+0000", 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.595438+0000", 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.595439+0000", 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.47e-07 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.595568+0000", 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0001294 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.595589+0000", 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0837999999999999e-05 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.595595+0000", 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6609999999999997e-06 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.595759+0000", 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016453 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.595790+0000", 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0023e-05 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.598490+0000", 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027004260000000001 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.598576+0000", 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5897999999999996e-05 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:73 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'51, mlcod=16'46)", 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.596416+0000", 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.501663485999998, 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032525309999999999, 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.596416+0000", 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.596416+0000", 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.596416+0000", 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.596422+0000", 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0869999999999997e-06 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.596438+0000", 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6676000000000001e-05 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.596443+0000", 2024-09-16T13:16:54.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7520000000000004e-06 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.596969+0000", 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052624900000000001 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.597041+0000", 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1372999999999996e-05 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.599603+0000", 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025623389999999998 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.599668+0000", 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5458999999999998e-05 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:74 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'52, mlcod=16'48)", 2024-09-16T13:16:54.144 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.597430+0000", 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.50064901, 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029273960000000001, 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.597430+0000", 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.597430+0000", 2024-09-16T13:16:54.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.597431+0000", 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9299999999999997e-07 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.597438+0000", 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2139999999999999e-06 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.597472+0000", 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3281999999999997e-05 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.597478+0000", 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7329999999999997e-06 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.597961+0000", 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048272499999999998 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.598001+0000", 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0252e-05 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.600296+0000", 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022949419999999999 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.600358+0000", 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1555000000000003e-05 2024-09-16T13:16:54.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:77 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'55, mlcod=16'53)", 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.602312+0000", 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.495766973999999, 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034289580000000002, 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602312+0000", 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602312+0000", 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602313+0000", 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0330000000000001e-06 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602379+0000", 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5840000000000007e-05 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602399+0000", 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9738999999999999e-05 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602406+0000", 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8929999999999999e-06 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602489+0000", 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3065000000000005e-05 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602535+0000", 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5728e-05 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605654+0000", 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031193480000000001 2024-09-16T13:16:54.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605741+0000", 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7311999999999993e-05 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:78 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'56, mlcod=16'53)", 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.602845+0000", 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.495234026999999, 2024-09-16T13:16:54.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002960598, 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602845+0000", 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602845+0000", 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602846+0000", 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.048e-06 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602906+0000", 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9867999999999997e-05 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602926+0000", 2024-09-16T13:16:54.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9306000000000001e-05 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602935+0000", 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7370000000000001e-06 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603371+0000", 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043586100000000001 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603399+0000", 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7551000000000001e-05 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605749+0000", 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023498939999999999 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605806+0000", 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7333000000000001e-05 2024-09-16T13:16:54.157 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:79 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'57, mlcod=16'53)", 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.603581+0000", 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.494498208, 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041617829999999996, 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603581+0000", 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603581+0000", 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603582+0000", 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603658+0000", 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5376999999999993e-05 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603729+0000", 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1766000000000004e-05 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603737+0000", 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6340000000000001e-06 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604193+0000", 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045645800000000003 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604248+0000", 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4242e-05 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.607658+0000", 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003410214 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.607743+0000", 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5092000000000005e-05 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:80 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.604358+0000", 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.493721139000002, 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060398550000000002, 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.162 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604358+0000", 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604358+0000", 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604359+0000", 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.054e-06 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604451+0000", 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1916000000000006e-05 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604469+0000", 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7376000000000001e-05 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604477+0000", 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0870000000000007e-06 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605124+0000", 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00064696399999999998 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605155+0000", 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1259e-05 2024-09-16T13:16:54.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.610343+0000", 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051882860000000003 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.610398+0000", 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4913000000000001e-05 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.166 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:89 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'63, mlcod=16'60)", 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:36.383688+0000", 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 17.714391190000001, 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028926109999999998, 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.383688+0000", 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.383688+0000", 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.383689+0000", 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1650000000000001e-06 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.383700+0000", 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.062e-05 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.383726+0000", 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6454000000000001e-05 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.383734+0000", 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0669999999999992e-06 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.383851+0000", 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116929 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.383911+0000", 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9330000000000003e-05 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.386510+0000", 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025988220000000002 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.386581+0000", 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1223999999999994e-05 2024-09-16T13:16:54.170 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:90 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'64, mlcod=16'60)", 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:36.385144+0000", 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 17.712935440999999, 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033364800000000002, 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385144+0000", 2024-09-16T13:16:54.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385144+0000", 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385146+0000", 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.643e-06 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385337+0000", 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019110199999999999 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385364+0000", 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7265999999999999e-05 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385375+0000", 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.147e-05 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385455+0000", 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9265999999999996e-05 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385481+0000", 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6752999999999999e-05 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388397+0000", 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029152079999999999 2024-09-16T13:16:54.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388480+0000", 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3771999999999997e-05 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:91 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'65, mlcod=16'60)", 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:36.387259+0000", 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 17.710820466000001, 2024-09-16T13:16:54.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038175589999999999, 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387259+0000", 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387259+0000", 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387260+0000", 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.082e-06 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387268+0000", 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7239999999999992e-06 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387284+0000", 2024-09-16T13:16:54.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6566999999999999e-05 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387293+0000", 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3599999999999996e-06 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387380+0000", 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7008000000000004e-05 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387419+0000", 2024-09-16T13:16:54.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9628000000000001e-05 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.390959+0000", 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003539964 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391076+0000", 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011722600000000001 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:92 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'66, mlcod=17'62)", 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:36.388758+0000", 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 17.709321280000001, 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032607000000000001, 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388758+0000", 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388758+0000", 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388759+0000", 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1319999999999999e-06 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388769+0000", 2024-09-16T13:16:54.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6560000000000004e-06 2024-09-16T13:16:54.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388792+0000", 2024-09-16T13:16:54.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3536e-05 2024-09-16T13:16:54.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388802+0000", 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388967+0000", 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016558999999999999 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.389011+0000", 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3451999999999997e-05 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391957+0000", 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00294655 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.392019+0000", 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1641000000000002e-05 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.184 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:94 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'68, mlcod=17'63)", 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:36.391209+0000", 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 17.706870476999999, 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003216455, 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391209+0000", 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391209+0000", 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391210+0000", 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.850000000000001e-07 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391218+0000", 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.909e-06 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.186 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391237+0000", 2024-09-16T13:16:54.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8748000000000001e-05 2024-09-16T13:16:54.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391244+0000", 2024-09-16T13:16:54.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4970000000000004e-06 2024-09-16T13:16:54.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391339+0000", 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4868e-05 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391419+0000", 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9932000000000007e-05 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.394363+0000", 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029436829999999999 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.394425+0000", 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2833e-05 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:96 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'70, mlcod=17'67)", 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:36.397712+0000", 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 17.700367781000001, 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033433479999999999, 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397712+0000", 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397712+0000", 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397713+0000", 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-06 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397789+0000", 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5776000000000001e-05 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397816+0000", 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7114000000000002e-05 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397833+0000", 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6748999999999998e-05 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397903+0000", 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9935e-05 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397930+0000", 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.794e-05 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.401013+0000", 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030821830000000001 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.401055+0000", 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2299000000000003e-05 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.194 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:54.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:16:54.196 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:54.186+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:16:54.297 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:54.292+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:16:54.303 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:16:54.313 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:54.313 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:16:54.314 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:16:54.314 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:54.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:16:54.398 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:54.390+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:54.413 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:54.414 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:16:54.414 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:16:54.414 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:16:54.414 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:54.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:16:54.499 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:54.490+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:16:54.522 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:54.522 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:16:54.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:16:54.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:16:54.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.522 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4281.0:5 2.2 e20/18 2:5cea7035:::rbd_info:head v 20'1, mlcod=0'0)", 2024-09-16T13:16:54.522 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:43.834287+0000", 2024-09-16T13:16:54.522 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.683093934, 2024-09-16T13:16:54.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001847521, 2024-09-16T13:16:54.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:43.834287+0000", 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:43.834287+0000", 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:43.834287+0000", 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9999999999999997e-07 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:43.834298+0000", 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0403e-05 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:43.834342+0000", 2024-09-16T13:16:54.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4069000000000001e-05 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:43.834358+0000", 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5945999999999999e-05 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:43.834529+0000", 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017117900000000001 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:43.834589+0000", 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0337000000000002e-05 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:43.836067+0000", 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001478023 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:43.836134+0000", 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6964000000000005e-05 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4281.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:44.841612+0000", 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.6757685020000004, 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021828479999999998, 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841612+0000", 2024-09-16T13:16:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841612+0000", 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841613+0000", 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841622+0000", 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.935e-06 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841645+0000", 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3686e-05 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841658+0000", 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2279999999999999e-05 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841760+0000", 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102268 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841830+0000", 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0000999999999996e-05 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.843740+0000", 2024-09-16T13:16:54.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019105789999999999 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.843795+0000", 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4691999999999997e-05 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:7 3.1a e25/23 3:5cea7035:::rbd_info:head v 25'1, mlcod=0'0)", 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:53.288535+0000", 2024-09-16T13:16:54.531 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.2288457660000001, 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001626288, 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:53.288535+0000", 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:53.288535+0000", 2024-09-16T13:16:54.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:53.288536+0000", 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4499999999999997e-07 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:53.288543+0000", 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6799999999999993e-06 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:53.288577+0000", 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3729000000000002e-05 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:53.288591+0000", 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:53.288674+0000", 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2727999999999998e-05 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:53.288749+0000", 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5240999999999993e-05 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:53.290100+0000", 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013500960000000001 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:53.290161+0000", 2024-09-16T13:16:54.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1746000000000001e-05 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:9 3.1a e26/23 3:5cea7035:::rbd_info:head v 26'2, mlcod=0'0)", 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.303946+0000", 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.21343446399999999, 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002066656, 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.303946+0000", 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.303946+0000", 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.303947+0000", 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:16:54.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.303954+0000", 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6090000000000003e-06 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.304014+0000", 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9966000000000002e-05 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.304022+0000", 2024-09-16T13:16:54.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5209999999999997e-06 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.304102+0000", 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0642999999999993e-05 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.304129+0000", 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.639e-05 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.305959+0000", 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018306329999999999 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.306013+0000", 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3514999999999998e-05 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:13 3.8 e26/23 3:12e71eb6:::rbd_object_map.112de98401a1:head v 26'1, mlcod=0'0)", 2024-09-16T13:16:54.540 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.315906+0000", 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.201475127, 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00095977300000000003, 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315906+0000", 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315906+0000", 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315906+0000", 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315913+0000", 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.148e-06 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315938+0000", 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5279999999999999e-05 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315948+0000", 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4129999999999995e-06 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316014+0000", 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5745000000000002e-05 2024-09-16T13:16:54.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316037+0000", 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3153999999999999e-05 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316811+0000", 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00077443300000000004 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316865+0000", 2024-09-16T13:16:54.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4156999999999997e-05 2024-09-16T13:16:54.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.545 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.545 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.545 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:54.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:16:54.600 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:54.592+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:16:54.642 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:54.643 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:16:54.643 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:16:54.643 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:54.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:16:54.700 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:54.696+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:54.744 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:54.744 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:16:54.744 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:16:54.744 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:16:54.745 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:54.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:16:54.801 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:54.796+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:54.847 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:54.847 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:16:54.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:16:54.847 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:16:54.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4281.0:1 2.3 e19/18 2:c4c92e5a:::rbd_trash:head v 19'1, mlcod=0'0)", 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:42.828375+0000", 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.015146469999999, 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00078740899999999998, 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:42.828375+0000", 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:42.828375+0000", 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:42.828376+0000", 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1699999999999997e-07 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:42.828383+0000", 2024-09-16T13:16:54.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4170000000000003e-06 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:42.828397+0000", 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3848e-05 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:42.828407+0000", 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9459999999999993e-06 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:54.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:42.828444+0000", 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6621000000000001e-05 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:42.828466+0000", 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2484e-05 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:42.829124+0000", 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000658437 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:54.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:42.829162+0000", 2024-09-16T13:16:54.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7938999999999997e-05 2024-09-16T13:16:54.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.852 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:54.852 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:54.852 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:54.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:16:54.902 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:54.896+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:54.951 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:54.951 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:16:54.951 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:16:54.952 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:54.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:16:55.003 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:54.996+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:55.053 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:55.054 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:16:55.054 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:16:55.054 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:16:55.054 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:55.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:16:55.103 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:55.094+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:55.160 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:55.160 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:16:55.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:16:55.160 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:16:55.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.160 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:12 3.2 3:4640f636:::rbd_header.112de98401a1:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.160 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.313311+0000", 2024-09-16T13:16:55.160 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.84053772599999999, 2024-09-16T13:16:55.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002018321, 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 12 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.313311+0000", 2024-09-16T13:16:55.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.313311+0000", 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.313313+0000", 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.313318+0000", 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5450000000000003e-06 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.313328+0000", 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6059999999999992e-06 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.313334+0000", 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5119999999999997e-06 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.313400+0000", 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5710000000000006e-05 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.313552+0000", 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015161800000000001 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.313747+0000", 2024-09-16T13:16:55.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000195207 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.314576+0000", 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082906699999999996 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315253+0000", 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00067654900000000003 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315295+0000", 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2932999999999997e-05 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315329+0000", 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3674999999999999e-05 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.4397.0:12 3.2 e26/23 ondisk, result = 0)", 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.315157+0000", 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.83869176899999998, 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018198300000000001, 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315157+0000", 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315157+0000", 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315158+0000", 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.33e-07 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315164+0000", 2024-09-16T13:16:55.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315194+0000", 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0199e-05 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315200+0000", 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4200000000000004e-06 2024-09-16T13:16:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315240+0000", 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9249999999999999e-05 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315245+0000", 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.852e-06 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315339+0000", 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3387999999999994e-05 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.315441+0000", 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.83840760199999997, 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001620844, 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.171 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315441+0000", 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315441+0000", 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315442+0000", 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2550000000000001e-06 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315448+0000", 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315462+0000", 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3383999999999999e-05 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315469+0000", 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8189999999999992e-06 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315525+0000", 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5730999999999999e-05 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315636+0000", 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110618 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.315793+0000", 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015744899999999999 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316555+0000", 2024-09-16T13:16:55.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00076202599999999996 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316987+0000", 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043196599999999999 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.317020+0000", 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.328e-05 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.317062+0000", 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1344000000000003e-05 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.4397.0:13 3.8 e26/23 ondisk, result = 0)", 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.316916+0000", 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.83693268700000001, 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015337799999999999, 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316916+0000", 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316916+0000", 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316917+0000", 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9899999999999999e-07 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316921+0000", 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1699999999999999e-06 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316932+0000", 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0801999999999999e-05 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316938+0000", 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6869999999999997e-06 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316976+0000", 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7991000000000003e-05 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.316983+0000", 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4570000000000002e-06 2024-09-16T13:16:55.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.317069+0000", 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6471999999999995e-05 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:16 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.321697+0000", 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.83215123800000002, 2024-09-16T13:16:55.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00064886499999999999, 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 16 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.321697+0000", 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.321697+0000", 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.321699+0000", 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4100000000000001e-06 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.321707+0000", 2024-09-16T13:16:55.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8869999999999992e-06 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.321721+0000", 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4459e-05 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.321727+0000", 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1140000000000002e-06 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.321823+0000", 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5981000000000001e-05 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.321994+0000", 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000170354 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322346+0000", 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035265999999999997 2024-09-16T13:16:55.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:17 3.2 3:4640f636:::rbd_header.112de98401a1:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.322494+0000", 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.83135426199999996, 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022547600000000001, 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.186 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 17 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322494+0000", 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322494+0000", 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322495+0000", 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5600000000000004e-07 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322501+0000", 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4729999999999999e-06 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322516+0000", 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5675000000000002e-05 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322523+0000", 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0990000000000004e-06 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322573+0000", 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0464999999999999e-05 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322630+0000", 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7354e-05 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322720+0000", 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9454000000000001e-05 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:18 3.2 3:4640f636:::rbd_header.112de98401a1:head [call,call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.322842+0000", 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.83100628899999995, 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00036766499999999998, 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 18 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322842+0000", 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322842+0000", 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322843+0000", 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0300000000000001e-06 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322849+0000", 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1199999999999999e-06 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322865+0000", 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5111e-05 2024-09-16T13:16:55.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322877+0000", 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2818e-05 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.322941+0000", 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4078000000000006e-05 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323033+0000", 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1349999999999998e-05 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323210+0000", 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017715799999999999 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:19 3.2 3:4640f636:::rbd_header.112de98401a1:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.195 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.323319+0000", 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.830529815, 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028321899999999997, 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 19 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323319+0000", 2024-09-16T13:16:55.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323319+0000", 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323320+0000", 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.158e-06 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323325+0000", 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.429e-06 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323341+0000", 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5614999999999999e-05 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323352+0000", 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1216000000000001e-05 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323398+0000", 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6063000000000003e-05 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323474+0000", 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5624999999999995e-05 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.323602+0000", 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012811300000000001 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.199 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:20 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.get_size in=11b,call rbd.get_features in=15b,call rbd.get_flags in=12b,call,call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.324150+0000", 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.82969811000000004, 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00062592699999999997, 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 20 2024-09-16T13:16:55.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324150+0000", 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324150+0000", 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324151+0000", 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3699999999999999e-07 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324157+0000", 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3680000000000001e-06 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324174+0000", 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7450999999999999e-05 2024-09-16T13:16:55.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324184+0000", 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0247e-05 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324244+0000", 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9818000000000002e-05 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324346+0000", 2024-09-16T13:16:55.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010186799999999999 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324776+0000", 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000430238 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:21 3.2 3:4640f636:::rbd_header.112de98401a1:head [call,call rbd.parent_overlap_get in=21b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.204 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.324910+0000", 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.82893899400000004, 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000393757, 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 21 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324910+0000", 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324910+0000", 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324911+0000", 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.043e-06 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324916+0000", 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6720000000000001e-06 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324933+0000", 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6364000000000001e-05 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324939+0000", 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5740000000000001e-06 2024-09-16T13:16:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.324980+0000", 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0351e-05 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325027+0000", 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7553999999999999e-05 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325303+0000", 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027619899999999998 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:22 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.325440+0000", 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.82840823500000005, 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000348282, 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.210 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 22 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325440+0000", 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325440+0000", 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325442+0000", 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.204e-06 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325447+0000", 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3700000000000003e-06 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325463+0000", 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6191000000000001e-05 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325476+0000", 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2503e-05 2024-09-16T13:16:55.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325531+0000", 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5658000000000001e-05 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325621+0000", 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9833999999999994e-05 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.214 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.325789+0000", 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000167522 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:24 3.2 3:4640f636:::rbd_header.112de98401a1:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.326782+0000", 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.82706613399999995, 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000354944, 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.215 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 24 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.326782+0000", 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.326782+0000", 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.326784+0000", 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.326790+0000", 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549999999999997e-06 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.326809+0000", 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8471e-05 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.326816+0000", 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0849999999999999e-06 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.218 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.326889+0000", 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2750000000000007e-05 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.326984+0000", 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5037999999999994e-05 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.327137+0000", 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015375799999999999 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.219 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:25 3.2 3:4640f636:::rbd_header.112de98401a1:head [watch watch cookie 140083176958464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.327656+0000", 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.82619293900000002, 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022961510000000002, 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:55.220 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 25 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.327656+0000", 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.327656+0000", 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.327657+0000", 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.147e-06 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.327662+0000", 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2059999999999998e-06 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.327678+0000", 2024-09-16T13:16:55.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6062e-05 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.327686+0000", 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4309999999999994e-06 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.327769+0000", 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2270999999999999e-05 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.327873+0000", 2024-09-16T13:16:55.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104713 2024-09-16T13:16:55.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:55.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.328174+0000", 2024-09-16T13:16:55.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030086400000000003 2024-09-16T13:16:55.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329047+0000", 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00087220799999999997 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329739+0000", 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069282000000000002 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:55.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329807+0000", 2024-09-16T13:16:55.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7576000000000005e-05 2024-09-16T13:16:55.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329952+0000", 2024-09-16T13:16:55.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014485299999999999 2024-09-16T13:16:55.226 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop_reply(client.4397.0:25 3.2 e26/23 ondisk, result = 0)", 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.329618+0000", 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.82423056299999997, 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033951000000000001, 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.227 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329618+0000", 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329618+0000", 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329619+0000", 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2640000000000001e-06 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329627+0000", 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2910000000000002e-06 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329645+0000", 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8090000000000001e-05 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:55.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329657+0000", 2024-09-16T13:16:55.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1922e-05 2024-09-16T13:16:55.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:55.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329716+0000", 2024-09-16T13:16:55.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9493000000000001e-05 2024-09-16T13:16:55.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:55.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329727+0000", 2024-09-16T13:16:55.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1026000000000001e-05 2024-09-16T13:16:55.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:55.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:55.232 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:55.232 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.329958+0000", 2024-09-16T13:16:55.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023042399999999999 2024-09-16T13:16:55.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:55.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:55.233 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:55.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:16:55.235 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:55.228+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:55.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:16:55.336 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:55.326+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:55.343 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:55.344 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:16:55.344 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:55.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:16:55.426 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:16:55.437 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:55.426+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:16:55.446 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:16:55.454 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:16:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:16:55.455 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:55.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:16:55.466 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:16:55.485 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:16:55.511 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:16:55.534 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:16:55.537 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:55.526+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:16:55.553 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:16:55.575 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:16:55.578 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:16:55.578 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:16:55.578 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:16:55.578 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:16:55.578 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:16:55.578 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:16:55.578 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:16:55.578 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:16:55.608 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:55.608 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:55.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:16:55.638 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:55.632+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:16:55.715 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:55.715 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:55.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:16:55.739 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:55.730+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:16:55.820 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:55.820 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:55.822 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:16:55.822 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:16:55.822 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T13:16:55.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T13:16:55.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:16:55.840 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:55.830+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:16:55.929 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:55.929 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:55.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:16:55.940 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:55.930+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:16:56.037 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:56.037 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:56.041 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:16:56.030+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:16:56.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:16:56.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 26, 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "6ddcae89-6c18-4eb7-8712-1c53f3b07278", 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-16T13:16:23.176689+0000", 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-16T13:16:54.280524+0000", 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "last_up_change": "2024-09-16T13:16:33.765413+0000", 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "last_in_change": "2024-09-16T13:16:25.201328+0000", 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_num": 5806080, 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_set": [ 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "nodeep-scrub", 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "noscrub", 2024-09-16T13:16:56.103 INFO:teuthology.orchestra.run.smithi098.stdout: "pglog_hardlimit", 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "purged_snapdirs", 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_deletes", 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "sortbitwise" 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_version": 5, 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_snapshot": "", 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_max": 3, 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd": 8, 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "require_min_compat_client": "luminous", 2024-09-16T13:16:56.104 INFO:teuthology.orchestra.run.smithi098.stdout: "min_compat_client": "jewel", 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "require_osd_release": "quincy", 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "pools": [ 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 1, 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": ".mgr", 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:33.497561+0000", 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool", 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:16:56.105 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 1, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 1, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 1, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 1, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 1, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:16:56.106 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "17", 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 0, 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 0, 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:16:56.107 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:16:56.108 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "options": { 2024-09-16T13:16:56.109 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_max": 32, 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_min": 1 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": {} 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 2, 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": "rbd", 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:40.994538+0000", 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 8193, 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:16:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 8, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 8, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 8, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 8, 2024-09-16T13:16:56.111 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 8, 2024-09-16T13:16:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:16:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:16:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:16:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:16:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:16:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:16:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:16:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.112 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "21", 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 3, 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 21, 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:16:56.113 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:16:56.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "options": {}, 2024-09-16T13:16:56.115 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd": {} 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 3, 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": "pool_client.0", 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:50.759705+0000", 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 8193, 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:16:56.116 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 32, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 32, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 32, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 32, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 32, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:16:56.117 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "26", 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 3, 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 26, 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:16:56.118 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:16:56.119 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:16:56.120 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "options": {}, 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd": {} 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "osds": [ 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 0, 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "be8543b4-3d28-4b37-a4ce-d2a56a09aeaf", 2024-09-16T13:16:56.121 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:16:56.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6808", 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6809", 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.123 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6810", 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6811", 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:16:56.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6816", 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6817", 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.125 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6813", 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6814", 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6809/22271", 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6811/22271", 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6817/22271", 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6814/22271", 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 1, 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "a7f95c7b-56d4-437b-a160-cabb8fd2b1e4", 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:16:56.127 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6812", 2024-09-16T13:16:56.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6815", 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6818", 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6819", 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6822", 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6823", 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6820", 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6821", 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6815/22272", 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6819/22272", 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6823/22272", 2024-09-16T13:16:56.132 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6821/22272", 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 2, 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c", 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:16:56.133 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 18, 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6800", 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6801", 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6802", 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6803", 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6806", 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:16:56.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6807", 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6804", 2024-09-16T13:16:56.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6805", 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6801/22273", 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6803/22273", 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6807/22273", 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6805/22273", 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:16:56.138 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 3, 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "4898b5b5-b912-430c-adcb-c52d90226917", 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:16:56.139 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6824", 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6825", 2024-09-16T13:16:56.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6826", 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6827", 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6830", 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6831", 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6828", 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6829", 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6825/22270", 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6827/22270", 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6831/22270", 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6829/22270", 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 4, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "6c478be2-f606-41d8-967c-9a689f09a440", 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:16:56.145 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6824", 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6825", 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6826", 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6827", 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.147 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6830", 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6831", 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.148 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6828", 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6829", 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6825/22286", 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6827/22286", 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6831/22286", 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6829/22286", 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 5, 2024-09-16T13:16:56.150 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "5001c828-b223-4d2c-b893-f9c56da73b83", 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6808", 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6809", 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6810", 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6811", 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6814", 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6815", 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6812", 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6813", 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6809/22285", 2024-09-16T13:16:56.155 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6811/22285", 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6815/22285", 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6813/22285", 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 6, 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "6f7a9125-674c-4be2-93e1-b6296cd00f8e", 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:16:56.156 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6816", 2024-09-16T13:16:56.157 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6817", 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6818", 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6819", 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6822", 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6823", 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6820", 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6821", 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6817/22315", 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6819/22315", 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6823/22315", 2024-09-16T13:16:56.161 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6821/22315", 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 7, 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "bcdfe435-f24d-45a9-a629-7380ad2e575a", 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:16:56.162 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6800", 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6801", 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6802", 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6803", 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6806", 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6807", 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6804", 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6805", 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6801/22284", 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6803/22284", 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6807/22284", 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6805/22284", 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_xinfo": [ 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 0, 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.191311+0000", 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.168 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 1, 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.563551+0000", 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 2, 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:16:56.169 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.493457+0000", 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 3, 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.599815+0000", 2024-09-16T13:16:56.170 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 4, 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.634352+0000", 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 5, 2024-09-16T13:16:56.171 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.262873+0000", 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 6, 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:16:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.508881+0000", 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 7, 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.348276+0000", 2024-09-16T13:16:56.173 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_upmap": [], 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_upmap_items": [], 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_temp": [], 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_temp": [], 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklist": {}, 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "range_blocklist": {}, 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profiles": { 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "default": { 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-failure-domain": "osd", 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "k": "2", 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "m": "1", 2024-09-16T13:16:56.174 INFO:teuthology.orchestra.run.smithi098.stdout: "plugin": "jerasure", 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "technique": "reed_sol_van" 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps_queue": [ 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 3, 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "snaps": [ 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "begin": 2, 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "length": 1 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:16:56.175 INFO:teuthology.orchestra.run.smithi098.stdout: "new_removed_snaps": [ 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 3, 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "snaps": [ 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "begin": 2, 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "length": 1 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "new_purged_snaps": [], 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_node_flags": {}, 2024-09-16T13:16:56.176 INFO:teuthology.orchestra.run.smithi098.stdout: "device_class_flags": {}, 2024-09-16T13:16:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": { 2024-09-16T13:16:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode_enabled": false, 2024-09-16T13:16:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_bucket_count": 0, 2024-09-16T13:16:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "degraded_stretch_mode": 0, 2024-09-16T13:16:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "recovering_stretch_mode": 0, 2024-09-16T13:16:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode_bucket": 0 2024-09-16T13:16:56.177 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:56.177 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:56.179 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:56.179 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:56.180 INFO:tasks.thrashosds.thrasher:j is {'epoch': 26, 'fsid': '6ddcae89-6c18-4eb7-8712-1c53f3b07278', 'created': '2024-09-16T13:16:23.176689+0000', 'modified': '2024-09-16T13:16:54.280524+0000', 'last_up_change': '2024-09-16T13:16:33.765413+0000', 'last_in_change': '2024-09-16T13:16:25.201328+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:16:33.497561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:16:40.994538+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-16T13:16:50.759705+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 26, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'be8543b4-3d28-4b37-a4ce-d2a56a09aeaf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6808', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6809', 'nonce': 22271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6810', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6811', 'nonce': 22271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6816', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6817', 'nonce': 22271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6813', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6814', 'nonce': 22271}]}, 'public_addr': '172.21.15.98:6809/22271', 'cluster_addr': '172.21.15.98:6811/22271', 'heartbeat_back_addr': '172.21.15.98:6817/22271', 'heartbeat_front_addr': '172.21.15.98:6814/22271', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a7f95c7b-56d4-437b-a160-cabb8fd2b1e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6812', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6815', 'nonce': 22272}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6818', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6819', 'nonce': 22272}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6822', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6823', 'nonce': 22272}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6820', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6821', 'nonce': 22272}]}, 'public_addr': '172.21.15.98:6815/22272', 'cluster_addr': '172.21.15.98:6819/22272', 'heartbeat_back_addr': '172.21.15.98:6823/22272', 'heartbeat_front_addr': '172.21.15.98:6821/22272', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6800', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6801', 'nonce': 22273}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6802', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6803', 'nonce': 22273}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6806', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6807', 'nonce': 22273}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6804', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6805', 'nonce': 22273}]}, 'public_addr': '172.21.15.98:6801/22273', 'cluster_addr': '172.21.15.98:6803/22273', 'heartbeat_back_addr': '172.21.15.98:6807/22273', 'heartbeat_front_addr': '172.21.15.98:6805/22273', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4898b5b5-b912-430c-adcb-c52d90226917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6824', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6825', 'nonce': 22270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6826', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6827', 'nonce': 22270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6830', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6831', 'nonce': 22270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6828', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6829', 'nonce': 22270}]}, 'public_addr': '172.21.15.98:6825/22270', 'cluster_addr': '172.21.15.98:6827/22270', 'heartbeat_back_addr': '172.21.15.98:6831/22270', 'heartbeat_front_addr': '172.21.15.98:6829/22270', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6c478be2-f606-41d8-967c-9a689f09a440', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 22286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 22286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 22286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 22286}]}, 'public_addr': '172.21.15.151:6825/22286', 'cluster_addr': '172.21.15.151:6827/22286', 'heartbeat_back_addr': '172.21.15.151:6831/22286', 'heartbeat_front_addr': '172.21.15.151:6829/22286', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5001c828-b223-4d2c-b893-f9c56da73b83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 22285}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 22285}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 22285}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 22285}]}, 'public_addr': '172.21.15.151:6809/22285', 'cluster_addr': '172.21.15.151:6811/22285', 'heartbeat_back_addr': '172.21.15.151:6815/22285', 'heartbeat_front_addr': '172.21.15.151:6813/22285', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6f7a9125-674c-4be2-93e1-b6296cd00f8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 22315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 22315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 22315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 22315}]}, 'public_addr': '172.21.15.151:6817/22315', 'cluster_addr': '172.21.15.151:6819/22315', 'heartbeat_back_addr': '172.21.15.151:6823/22315', 'heartbeat_front_addr': '172.21.15.151:6821/22315', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bcdfe435-f24d-45a9-a629-7380ad2e575a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6800', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6801', 'nonce': 22284}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 22284}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 22284}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 22284}]}, 'public_addr': '172.21.15.151:6801/22284', 'cluster_addr': '172.21.15.151:6803/22284', 'heartbeat_back_addr': '172.21.15.151:6807/22284', 'heartbeat_front_addr': '172.21.15.151:6805/22284', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.191311+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.563551+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.493457+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.599815+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.634352+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.262873+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.508881+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.348276+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T13:16:56.180 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-16T13:16:56.180 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:56.172+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:16:56.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:16:56.243 INFO:teuthology.orchestra.run.smithi098.stdout:truncating to largest ever: 0x51ca095 2024-09-16T13:16:56.251 INFO:teuthology.orchestra.run.smithi098.stdout:1 trunc from 0x0 to 0x51ca095 2024-09-16T13:16:56.262 INFO:teuthology.orchestra.run.smithi098.stdout:3 read 0x1422bbe thru 0x142b166 (0x85a9 bytes) 2024-09-16T13:16:56.262 INFO:teuthology.orchestra.run.smithi098.stdout:4 punch from 0x494865 to 0x4989fa, (0x4195 bytes) 2024-09-16T13:16:56.276 INFO:teuthology.orchestra.run.smithi098.stdout:5 writesame 0x5dad3bf thru 0x5db0eb5 data_size 0x1(0x3af7 bytes) 2024-09-16T13:16:56.276 INFO:teuthology.orchestra.run.smithi098.stdout:6 punch from 0x1174c38 to 0x117d7a6, (0x8b6e bytes) 2024-09-16T13:16:56.276 INFO:teuthology.orchestra.run.smithi098.stdout:9 write 0x27156cd thru 0x2724619 (0xef4d bytes) 2024-09-16T13:16:56.279 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:56.272+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:56.282 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:56.282 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:56.282 INFO:teuthology.orchestra.run.smithi098.stdout:truncating to largest ever: 0xa0325e2 2024-09-16T13:16:56.287 INFO:teuthology.orchestra.run.smithi098.stdout:10 trunc from 0x5db0eb6 to 0xa0325e2 2024-09-16T13:16:56.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:16:56.307 INFO:teuthology.orchestra.run.smithi098.stdout:12 trunc from 0xa0325e2 to 0x5c643b4 2024-09-16T13:16:56.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:16:56.317 INFO:teuthology.orchestra.run.smithi098.stdout:13 read 0x2a02e0 thru 0x2a62c9 (0x5fea bytes) 2024-09-16T13:16:56.342 INFO:teuthology.orchestra.run.smithi098.stdout:14 write 0xddc1209 thru 0xddd044e (0xf246 bytes) 2024-09-16T13:16:56.349 INFO:teuthology.orchestra.run.smithi098.stdout:16 writesame 0x7dd81b9 thru 0x7de0894 data_size 0x2a(0x86dc bytes) 2024-09-16T13:16:56.356 INFO:teuthology.orchestra.run.smithi098.stdout:17 write 0x37e50e5 thru 0x37f1855 (0xc771 bytes) 2024-09-16T13:16:56.362 INFO:teuthology.orchestra.run.smithi098.stdout:18 writesame 0xab5af00 thru 0xab5d96b data_size 0x1e(0x2a6c bytes) 2024-09-16T13:16:56.369 INFO:teuthology.orchestra.run.smithi098.stdout:19 read 0x529a4ad thru 0x52a76c4 (0xd218 bytes) 2024-09-16T13:16:56.369 INFO:teuthology.orchestra.run.smithi098.stdout:20 read 0x7fb411d thru 0x7fba34b (0x622f bytes) 2024-09-16T13:16:56.370 INFO:teuthology.orchestra.run.smithi098.stdout:21 read 0xeff23f thru 0xf0ae69 (0xbc2b bytes) 2024-09-16T13:16:56.370 INFO:teuthology.orchestra.run.smithi098.stdout:22 write 0x3eb838 thru 0x3ec1d1 (0x99a bytes) 2024-09-16T13:16:56.375 INFO:teuthology.orchestra.run.smithi098.stdout:23 writesame 0x6122c2f thru 0x612cc73 data_size 0x1(0xa045 bytes) 2024-09-16T13:16:56.380 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:56.370+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:16:56.397 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:56.397 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:56.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:16:56.422 INFO:teuthology.orchestra.run.smithi098.stdout:24 read 0x8ab9373 thru 0x8ac2680 (0x930e bytes) 2024-09-16T13:16:56.422 INFO:teuthology.orchestra.run.smithi098.stdout:25 read 0x207e60a thru 0x208139c (0x2d93 bytes) 2024-09-16T13:16:56.429 INFO:teuthology.orchestra.run.smithi098.stdout:26 trunc from 0xddd044f to 0x9242c9a 2024-09-16T13:16:56.448 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:56.459 INFO:teuthology.orchestra.run.smithi098.stdout:28 writesame 0xa53978b thru 0xa53d9b2 data_size 0x3a(0x4228 bytes) 2024-09-16T13:16:56.460 INFO:teuthology.orchestra.run.smithi098.stdout:29 writesame 0x5c6a4bb thru 0x5c700b1 data_size 0x1(0x5bf7 bytes) 2024-09-16T13:16:56.481 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:56.476+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:56.499 INFO:teuthology.orchestra.run.smithi098.stdout:30 write 0xcd23772 thru 0xcd23a65 (0x2f4 bytes) 2024-09-16T13:16:56.500 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:56.500 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:56.504 INFO:teuthology.orchestra.run.smithi098.stdout:31 write 0xc41a9c5 thru 0xc4264cd (0xbb09 bytes) 2024-09-16T13:16:56.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:16:56.510 INFO:teuthology.orchestra.run.smithi098.stdout:truncating to largest ever: 0xd8832fa 2024-09-16T13:16:56.511 INFO:teuthology.orchestra.run.smithi098.stdout:32 trunc from 0xcd23a66 to 0xd8832fa 2024-09-16T13:16:56.525 INFO:teuthology.orchestra.run.smithi098.stdout:33 writesame 0x78ef8c8 thru 0x78fb5a3 data_size 0x66(0xbcdc bytes) 2024-09-16T13:16:56.531 INFO:teuthology.orchestra.run.smithi098.stdout:35 punch from 0x4ba611e to 0x4bad7d4, (0x76b6 bytes) 2024-09-16T13:16:56.540 INFO:teuthology.orchestra.run.smithi098.stdout:36 trunc from 0xd8832fa to 0x7978699 2024-09-16T13:16:56.570 INFO:teuthology.orchestra.run.smithi098.stdout:37 trunc from 0x7978699 to 0xd7bbb76 2024-09-16T13:16:56.582 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:56.576+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:56.590 INFO:teuthology.orchestra.run.smithi098.stdout:38 writesame 0xe1fc423 thru 0xe2051b7 data_size 0xb(0x8d95 bytes) 2024-09-16T13:16:56.590 INFO:teuthology.orchestra.run.smithi098.stdout:39 writesame 0xaefdcf9 thru 0xaefdd73 data_size 0x1(0x7b bytes) 2024-09-16T13:16:56.596 INFO:teuthology.orchestra.run.smithi098.stdout:41 punch from 0x7f33130 to 0x7f3d3c3, (0xa293 bytes) 2024-09-16T13:16:56.596 INFO:teuthology.orchestra.run.smithi098.stdout:42 read 0x9e10c7f thru 0x9e13c7a (0x2ffc bytes) 2024-09-16T13:16:56.596 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:56.596 INFO:teuthology.orchestra.run.smithi098.stdout:45 writesame 0xd2f2775 thru 0xd2fd1dd data_size 0x5(0xaa69 bytes) 2024-09-16T13:16:56.606 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:56.606 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:56.609 INFO:teuthology.orchestra.run.smithi098.stdout:47 punch from 0x81b7f83 to 0x81c7752, (0xf7cf bytes) 2024-09-16T13:16:56.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:16:56.618 INFO:teuthology.orchestra.run.smithi098.stdout:49 writesame 0xe5352f3 thru 0xe53fe75 data_size 0x17(0xab83 bytes) 2024-09-16T13:16:56.618 INFO:teuthology.orchestra.run.smithi098.stdout:50 write 0x7b6c94f thru 0x7b6dc67 (0x1319 bytes) 2024-09-16T13:16:56.619 INFO:teuthology.orchestra.run.smithi098.stdout:51 read 0xa40692 thru 0xa4a71a (0xa089 bytes) 2024-09-16T13:16:56.630 INFO:teuthology.orchestra.run.smithi098.stdout:52 writesame 0xecf742a thru 0xed04501 data_size 0x8(0xd0d8 bytes) 2024-09-16T13:16:56.630 INFO:teuthology.orchestra.run.smithi098.stdout:53 write 0xb45599e thru 0xb4651f4 (0xf857 bytes) 2024-09-16T13:16:56.636 INFO:teuthology.orchestra.run.smithi098.stdout:54 write 0xc5b4bc6 thru 0xc5c0598 (0xb9d3 bytes) 2024-09-16T13:16:56.641 INFO:teuthology.orchestra.run.smithi098.stdout:55 write 0xb438a42 thru 0xb447a6c (0xf02b bytes) 2024-09-16T13:16:56.666 INFO:teuthology.orchestra.run.smithi098.stdout:56 trunc from 0xed04502 to 0x64345 2024-09-16T13:16:56.683 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:56.674+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:16:56.688 INFO:teuthology.orchestra.run.smithi098.stdout:57 read 0x59272 thru 0x64344 (0xb0d3 bytes) 2024-09-16T13:16:56.689 INFO:teuthology.orchestra.run.smithi098.stdout:58 punch from 0x26e2b to 0x335ab, (0xc780 bytes) 2024-09-16T13:16:56.689 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:56.689 INFO:teuthology.orchestra.run.smithi098.stdout:60 read 0x48196 thru 0x489e7 (0x852 bytes) 2024-09-16T13:16:56.690 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:56.703 INFO:teuthology.orchestra.run.smithi098.stdout:62 write 0x28a99ec thru 0x28adfd2 (0x45e7 bytes) 2024-09-16T13:16:56.716 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:56.716 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:56.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:16:56.728 INFO:teuthology.orchestra.run.smithi098.stdout:63 write 0x879f0da thru 0x87a1bee (0x2b15 bytes) 2024-09-16T13:16:56.735 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:56.735 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:56.735 INFO:teuthology.orchestra.run.smithi098.stdout:69 punch from 0x50334ad to 0x504158b, (0xe0de bytes) 2024-09-16T13:16:56.735 INFO:teuthology.orchestra.run.smithi098.stdout:72 read 0x3596f99 thru 0x359f363 (0x83cb bytes) 2024-09-16T13:16:56.735 INFO:teuthology.orchestra.run.smithi098.stdout:73 read 0x81d84e2 thru 0x81e2bac (0xa6cb bytes) 2024-09-16T13:16:56.751 INFO:teuthology.orchestra.run.smithi098.stdout:74 write 0xbf2729a thru 0xbf2b031 (0x3d98 bytes) 2024-09-16T13:16:56.757 INFO:teuthology.orchestra.run.smithi098.stdout:75 writesame 0x86ce7aa thru 0x86d5f57 data_size 0x1(0x77ae bytes) 2024-09-16T13:16:56.784 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:56.774+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:16:56.806 INFO:teuthology.orchestra.run.smithi098.stdout:76 write 0xed67c91 thru 0xed7368f (0xb9ff bytes) 2024-09-16T13:16:56.812 INFO:teuthology.orchestra.run.smithi098.stdout:77 write 0x686d9a2 thru 0x6876be2 (0x9241 bytes) 2024-09-16T13:16:56.818 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:56.818 INFO:teuthology.orchestra.run.smithi098.stdout:79 writesame 0x5541b6a thru 0x5548e97 data_size 0x17(0x732e bytes) 2024-09-16T13:16:56.825 INFO:teuthology.orchestra.run.smithi098.stdout:80 write 0xde78d5d thru 0xde8681a (0xdabe bytes) 2024-09-16T13:16:56.833 INFO:teuthology.orchestra.run.smithi098.stdout:81 write 0x4f5c187 thru 0x4f6a8a6 (0xe720 bytes) 2024-09-16T13:16:56.833 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:56.833 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:56.840 INFO:teuthology.orchestra.run.smithi098.stdout:82 write 0x8e54811 thru 0x8e5c44e (0x7c3e bytes) 2024-09-16T13:16:56.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:16:56.848 INFO:teuthology.orchestra.run.smithi098.stdout:84 read 0x300001a thru 0x3001af7 (0x1ade bytes) 2024-09-16T13:16:56.848 INFO:teuthology.orchestra.run.smithi098.stdout:85 read 0xafbeee8 thru 0xafc5c1b (0x6d34 bytes) 2024-09-16T13:16:56.867 INFO:teuthology.orchestra.run.smithi098.stdout:86 trunc from 0xed73690 to 0x3796d68 2024-09-16T13:16:56.885 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:56.874+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:16:56.893 INFO:teuthology.orchestra.run.smithi098.stdout:88 read 0x14f2126 thru 0x14fb6ce (0x95a9 bytes) 2024-09-16T13:16:56.894 INFO:teuthology.orchestra.run.smithi098.stdout:90 punch from 0x2e0a3c7 to 0x2e0c5cd, (0x2206 bytes) 2024-09-16T13:16:56.920 INFO:teuthology.orchestra.run.smithi098.stdout:91 write 0xcc7a30b thru 0xcc7b0c1 (0xdb7 bytes) 2024-09-16T13:16:56.926 INFO:teuthology.orchestra.run.smithi098.stdout:94 write 0x330d6ea thru 0x3316f4e (0x9865 bytes) 2024-09-16T13:16:56.932 INFO:teuthology.orchestra.run.smithi098.stdout:95 writesame 0x5bf3fda thru 0x5c00330 data_size 0x3(0xc357 bytes) 2024-09-16T13:16:56.952 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:56.952 INFO:teuthology.orchestra.run.smithi098.stdout:98 punch from 0x3d38830 to 0x3d47653, (0xee23 bytes) 2024-09-16T13:16:56.960 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:56.960 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:56.963 INFO:teuthology.orchestra.run.smithi098.stdout:100 trunc from 0xcc7b0c2 to 0x733e7df 2024-09-16T13:16:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:16:56.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:16:56.991 INFO:teuthology.orchestra.run.smithi098.stdout:101 writesame 0x54d6adb thru 0x54deaa6 data_size 0x2(0x7fcc bytes) 2024-09-16T13:16:56.992 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:56.984+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:16:57.018 INFO:teuthology.orchestra.run.smithi098.stdout:102 trunc from 0x733e7df to 0x21a7b59 2024-09-16T13:16:57.053 INFO:teuthology.orchestra.run.smithi098.stdout:104 write 0x3d32efe thru 0x3d39c50 (0x6d53 bytes) 2024-09-16T13:16:57.060 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.086 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:57.086 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:57.087 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:57.078+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:57.088 INFO:teuthology.orchestra.run.smithi098.stdout:106 write 0xcccb3cf thru 0xccd5a9e (0xa6d0 bytes) 2024-09-16T13:16:57.094 INFO:teuthology.orchestra.run.smithi098.stdout:107 write 0x8305e3f thru 0x83152fd (0xf4bf bytes) 2024-09-16T13:16:57.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:16:57.101 INFO:teuthology.orchestra.run.smithi098.stdout:108 writesame 0x1faad45 thru 0x1fb1e11 data_size 0x1(0x70cd bytes) 2024-09-16T13:16:57.127 INFO:teuthology.orchestra.run.smithi098.stdout:109 punch from 0x898f318 to 0x899e362, (0xf04a bytes) 2024-09-16T13:16:57.138 INFO:teuthology.orchestra.run.smithi098.stdout:110 trunc from 0xccd5a9f to 0x5d11ba9 2024-09-16T13:16:57.158 INFO:teuthology.orchestra.run.smithi098.stdout:111 write 0x2046e1c thru 0x204d5b3 (0x6798 bytes) 2024-09-16T13:16:57.165 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.165 INFO:teuthology.orchestra.run.smithi098.stdout:113 write 0x43b54b0 thru 0x43ba036 (0x4b87 bytes) 2024-09-16T13:16:57.172 INFO:teuthology.orchestra.run.smithi098.stdout:114 punch from 0x4637ad2 to 0x4638178, (0x6a6 bytes) 2024-09-16T13:16:57.172 INFO:teuthology.orchestra.run.smithi098.stdout:115 write 0x7a642f thru 0x7ae5e3 (0x81b5 bytes) 2024-09-16T13:16:57.178 INFO:teuthology.orchestra.run.smithi098.stdout:116 read 0xb450c8 thru 0xb50e25 (0xbd5e bytes) 2024-09-16T13:16:57.178 INFO:teuthology.orchestra.run.smithi098.stdout:117 read 0x4f1df2a thru 0x4f29490 (0xb567 bytes) 2024-09-16T13:16:57.178 INFO:teuthology.orchestra.run.smithi098.stdout:118 read 0x167a734 thru 0x167abf7 (0x4c4 bytes) 2024-09-16T13:16:57.188 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:57.178+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:57.203 INFO:teuthology.orchestra.run.smithi098.stdout:120 write 0xdff6a2b thru 0xdffae13 (0x43e9 bytes) 2024-09-16T13:16:57.207 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:57.207 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:57.210 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.211 INFO:teuthology.orchestra.run.smithi098.stdout:123 read 0xcfc3a7 thru 0xd05d55 (0x99af bytes) 2024-09-16T13:16:57.211 INFO:teuthology.orchestra.run.smithi098.stdout:125 write 0xc981be3 thru 0xc985272 (0x3690 bytes) 2024-09-16T13:16:57.217 INFO:teuthology.orchestra.run.smithi098.stdout:126 punch from 0x2a92261 to 0x2a9a05f, (0x7dfe bytes) 2024-09-16T13:16:57.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:16:57.229 INFO:teuthology.orchestra.run.smithi098.stdout:127 write 0xea40bc2 thru 0xea4e919 (0xdd58 bytes) 2024-09-16T13:16:57.236 INFO:teuthology.orchestra.run.smithi098.stdout:128 write 0xa3f93a thru 0xa4ceac (0xd573 bytes) 2024-09-16T13:16:57.243 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.243 INFO:teuthology.orchestra.run.smithi098.stdout:131 read 0x727114e thru 0x727e721 (0xd5d4 bytes) 2024-09-16T13:16:57.244 INFO:teuthology.orchestra.run.smithi098.stdout:133 writesame 0x4117336 thru 0x4121a82 data_size 0x1(0xa74d bytes) 2024-09-16T13:16:57.283 INFO:teuthology.orchestra.run.smithi098.stdout:134 read 0x728a476 thru 0x728f8a9 (0x5434 bytes) 2024-09-16T13:16:57.284 INFO:teuthology.orchestra.run.smithi098.stdout:135 writesame 0x891ff36 thru 0x89298e0 data_size 0x9(0x99ab bytes) 2024-09-16T13:16:57.289 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:57.278+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:57.293 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.293 INFO:teuthology.orchestra.run.smithi098.stdout:137 read 0x85cd076 thru 0x85daed8 (0xde63 bytes) 2024-09-16T13:16:57.293 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.294 INFO:teuthology.orchestra.run.smithi098.stdout:139 write 0xccc096a thru 0xccc7c7c (0x7313 bytes) 2024-09-16T13:16:57.300 INFO:teuthology.orchestra.run.smithi098.stdout:140 write 0xb87d1ca thru 0xb88701d (0x9e54 bytes) 2024-09-16T13:16:57.306 INFO:teuthology.orchestra.run.smithi098.stdout:142 write 0xe3ab2fe thru 0xe3bb263 (0xff66 bytes) 2024-09-16T13:16:57.313 INFO:teuthology.orchestra.run.smithi098.stdout:144 write 0xd8b891b thru 0xd8c2f8e (0xa674 bytes) 2024-09-16T13:16:57.320 INFO:teuthology.orchestra.run.smithi098.stdout:145 read 0xaef0662 thru 0xaef35ba (0x2f59 bytes) 2024-09-16T13:16:57.320 INFO:teuthology.orchestra.run.smithi098.stdout:149 writesame 0xb891a4 thru 0xb98213 data_size 0x8(0xf070 bytes) 2024-09-16T13:16:57.329 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:57.330 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:57.330 INFO:teuthology.orchestra.run.smithi098.stdout:150 write 0x802e87 thru 0x8083d8 (0x5552 bytes) 2024-09-16T13:16:57.335 INFO:teuthology.orchestra.run.smithi098.stdout:151 read 0x4f2651a thru 0x4f307db (0xa2c2 bytes) 2024-09-16T13:16:57.335 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.335 INFO:teuthology.orchestra.run.smithi098.stdout:153 read 0x537a058 thru 0x5385641 (0xb5ea bytes) 2024-09-16T13:16:57.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:16:57.341 INFO:teuthology.orchestra.run.smithi098.stdout:154 trunc from 0xea4e91a to 0xadca7c2 2024-09-16T13:16:57.361 INFO:teuthology.orchestra.run.smithi098.stdout:156 writesame 0x84ac3eb thru 0x84b0b9d data_size 0x1(0x47b3 bytes) 2024-09-16T13:16:57.380 INFO:teuthology.orchestra.run.smithi098.stdout:157 write 0x4bed288 thru 0x4bee0a6 (0xe1f bytes) 2024-09-16T13:16:57.390 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:57.382+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:16:57.400 INFO:teuthology.orchestra.run.smithi098.stdout:158 trunc from 0xadca7c2 to 0x2e5a528 2024-09-16T13:16:57.422 INFO:teuthology.orchestra.run.smithi098.stdout:159 writesame 0x115bd51 thru 0x115ed71 data_size 0x25(0x3021 bytes) 2024-09-16T13:16:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:163 write 0x14d7e05 thru 0x14de0b9 (0x62b5 bytes) 2024-09-16T13:16:57.433 INFO:teuthology.orchestra.run.smithi098.stdout:164 read 0x382dba thru 0x38927a (0x64c1 bytes) 2024-09-16T13:16:57.434 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.434 INFO:teuthology.orchestra.run.smithi098.stdout:166 punch from 0x1358b72 to 0x1366c0f, (0xe09d bytes) 2024-09-16T13:16:57.434 INFO:teuthology.orchestra.run.smithi098.stdout:169 read 0xc483e4 thru 0xc4d5f1 (0x520e bytes) 2024-09-16T13:16:57.434 INFO:teuthology.orchestra.run.smithi098.stdout:171 punch from 0xecffcd to 0xed545d, (0x5490 bytes) 2024-09-16T13:16:57.449 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:57.449 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:57.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:16:57.465 INFO:teuthology.orchestra.run.smithi098.stdout:172 writesame 0xe3b795f thru 0xe3b8fda data_size 0x2(0x167c bytes) 2024-09-16T13:16:57.465 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.465 INFO:teuthology.orchestra.run.smithi098.stdout:175 write 0xaba6d87 thru 0xabaca12 (0x5c8c bytes) 2024-09-16T13:16:57.469 INFO:teuthology.orchestra.run.smithi098.stdout:176 read 0x91f0f55 thru 0x91fa623 (0x96cf bytes) 2024-09-16T13:16:57.469 INFO:teuthology.orchestra.run.smithi098.stdout:178 write 0xca3eaf3 thru 0xca46734 (0x7c42 bytes) 2024-09-16T13:16:57.476 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.482 INFO:teuthology.orchestra.run.smithi098.stdout:183 trunc from 0xe3b8fdb to 0xac75e0b 2024-09-16T13:16:57.490 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:57.482+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:57.504 INFO:teuthology.orchestra.run.smithi098.stdout:184 punch from 0x89b7875 to 0x89c4b17, (0xd2a2 bytes) 2024-09-16T13:16:57.504 INFO:teuthology.orchestra.run.smithi098.stdout:185 write 0x87180f0 thru 0x8724893 (0xc7a4 bytes) 2024-09-16T13:16:57.510 INFO:teuthology.orchestra.run.smithi098.stdout:186 writesame 0x77f0323 thru 0x77ff0ea data_size 0x1c(0xedc8 bytes) 2024-09-16T13:16:57.518 INFO:teuthology.orchestra.run.smithi098.stdout:187 punch from 0x67afdd8 to 0x67bf86d, (0xfa95 bytes) 2024-09-16T13:16:57.518 INFO:teuthology.orchestra.run.smithi098.stdout:189 read 0x1bd9b47 thru 0x1be4793 (0xac4d bytes) 2024-09-16T13:16:57.519 INFO:teuthology.orchestra.run.smithi098.stdout:193 read 0x1cc1fd6 thru 0x1cca18f (0x81ba bytes) 2024-09-16T13:16:57.519 INFO:teuthology.orchestra.run.smithi098.stdout:194 punch from 0x7ada665 to 0x7ae770b, (0xd0a6 bytes) 2024-09-16T13:16:57.519 INFO:teuthology.orchestra.run.smithi098.stdout:196 write 0x74fa0af thru 0x750a0ae (0x10000 bytes) 2024-09-16T13:16:57.525 INFO:teuthology.orchestra.run.smithi098.stdout:197 read 0x32ec87a thru 0x32fc050 (0xf7d7 bytes) 2024-09-16T13:16:57.543 INFO:teuthology.orchestra.run.smithi098.stdout:198 trunc from 0xac75e0b to 0xea6df5 2024-09-16T13:16:57.565 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:57.565 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:57.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:16:57.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:16:57.591 INFO:teuthology.orchestra.run.smithi098.stdout:199 write 0x54b941b thru 0x54c855a (0xf140 bytes) 2024-09-16T13:16:57.591 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:57.584+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:57.597 INFO:teuthology.orchestra.run.smithi098.stdout:200 punch from 0x2326fff to 0x232e1ea, (0x71eb bytes) 2024-09-16T13:16:57.597 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.597 INFO:teuthology.orchestra.run.smithi098.stdout:205 read 0x4576df9 thru 0x458614a (0xf352 bytes) 2024-09-16T13:16:57.597 INFO:teuthology.orchestra.run.smithi098.stdout:206 read 0x4c17651 thru 0x4c21b56 (0xa506 bytes) 2024-09-16T13:16:57.622 INFO:teuthology.orchestra.run.smithi098.stdout:208 writesame 0xcb8e93e thru 0xcb93199 data_size 0x16(0x485c bytes) 2024-09-16T13:16:57.622 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.639 INFO:teuthology.orchestra.run.smithi098.stdout:210 trunc from 0xcb9319a to 0x2883592 2024-09-16T13:16:57.656 INFO:teuthology.orchestra.run.smithi098.stdout:211 read 0x24c78e8 thru 0x24d74a6 (0xfbbf bytes) 2024-09-16T13:16:57.656 INFO:teuthology.orchestra.run.smithi098.stdout:212 read 0x24fb88b thru 0x24fcf7b (0x16f1 bytes) 2024-09-16T13:16:57.656 INFO:teuthology.orchestra.run.smithi098.stdout:213 write 0x13f956e thru 0x1403536 (0x9fc9 bytes) 2024-09-16T13:16:57.664 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.665 INFO:teuthology.orchestra.run.smithi098.stdout:215 writesame 0xaf0f66 thru 0xaf93a0 data_size 0x1(0x843b bytes) 2024-09-16T13:16:57.692 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:57.682+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:57.699 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:16:57.699 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:16:57.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:16:57.714 INFO:teuthology.orchestra.run.smithi098.stdout:216 write 0x54a471e thru 0x54a719f (0x2a82 bytes) 2024-09-16T13:16:57.719 INFO:teuthology.orchestra.run.smithi098.stdout:217 punch from 0x43e8930 to 0x43f5f44, (0xd614 bytes) 2024-09-16T13:16:57.719 INFO:teuthology.orchestra.run.smithi098.stdout:truncating to largest ever: 0xe5bbd2f 2024-09-16T13:16:57.734 INFO:teuthology.orchestra.run.smithi098.stdout:218 trunc from 0x54a71a0 to 0xe5bbd2f 2024-09-16T13:16:57.738 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:16:57.744 INFO:teuthology.orchestra.run.smithi098.stdout:220 read 0x688d301 thru 0x689c615 (0xf315 bytes) 2024-09-16T13:16:57.744 INFO:teuthology.orchestra.run.smithi098.stdout:221 writesame 0x9cdcd7d thru 0x9ce386b data_size 0x19(0x6aef bytes) 2024-09-16T13:16:57.756 INFO:teuthology.orchestra.run.smithi098.stdout:222 trunc from 0xe5bbd2f to 0xb5316fd 2024-09-16T13:16:57.769 INFO:teuthology.orchestra.run.smithi098.stdout:223 trunc from 0xb5316fd to 0xbf9a726 2024-09-16T13:16:57.771 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:16:57.780 INFO:teuthology.orchestra.run.smithi098.stdout:225 read 0xaba26ae thru 0xabaf758 (0xd0ab bytes) 2024-09-16T13:16:57.780 INFO:teuthology.orchestra.run.smithi098.stdout:226 read 0x3e99c01 thru 0x3ea83f7 (0xe7f7 bytes) 2024-09-16T13:16:57.780 INFO:teuthology.orchestra.run.smithi098.stdout:227 write 0x31d5549 thru 0x31da7ed (0x52a5 bytes) 2024-09-16T13:16:57.786 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:57.787 INFO:teuthology.orchestra.run.smithi098.stdout:229 trunc from 0xbf9a726 to 0xcb0c11b 2024-09-16T13:16:57.792 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:57.788+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:57.793 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:16:57.797 INFO:teuthology.orchestra.run.smithi098.stdout:231 writesame 0xb87d9b1 thru 0xb883818 data_size 0x72(0x5e68 bytes) 2024-09-16T13:16:57.802 INFO:teuthology.orchestra.run.smithi098.stdout:232 punch from 0x542f03d to 0x543c8c4, (0xd887 bytes) 2024-09-16T13:16:57.802 INFO:teuthology.orchestra.run.smithi098.stdout:233 write 0x7af3263 thru 0x7afc31a (0x90b8 bytes) 2024-09-16T13:16:57.807 INFO:teuthology.orchestra.run.smithi098.stdout:234 read 0x613c287 thru 0x61435a7 (0x7321 bytes) 2024-09-16T13:16:57.813 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:16:57.817 INFO:teuthology.orchestra.run.smithi098.stdout:235 trunc from 0xcb0c11b to 0x7381e27 2024-09-16T13:16:57.831 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:57.831 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:16:57.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:653 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.get_size in=19b out=9b,call rbd.get_features in=24b out=16b,call rbd.get_flags in=20b out=8b,call rbd.get_snapcontext in=18b,call lock.get_info in=30b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.827314+0000", 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00032511999999999998, 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000366936, 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 653 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:57.832 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.827314+0000", 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.827314+0000", 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.827315+0000", 2024-09-16T13:16:57.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3799999999999996e-07 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.827320+0000", 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1139999999999997e-06 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.827335+0000", 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4717e-05 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:57.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.827342+0000", 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7360000000000004e-06 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.827389+0000", 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7403e-05 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.827472+0000", 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2360000000000004e-05 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:57.835 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:57.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:57.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:57.836 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:16:57.836 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 1 2024-09-16T13:16:57.836 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:57.840 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:16:57.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:16:57.847 INFO:teuthology.orchestra.run.smithi098.stdout:238 write 0x9dc3b2a thru 0x9dcbb58 (0x802f bytes) 2024-09-16T13:16:57.863 INFO:teuthology.orchestra.run.smithi098.stdout:240 write 0xa735af6 thru 0xa73925f (0x376a bytes) 2024-09-16T13:16:57.863 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:16:57.872 INFO:teuthology.orchestra.run.smithi098.stdout:241 punch from 0x88db2f0 to 0x88e8795, (0xd4a5 bytes) 2024-09-16T13:16:57.872 INFO:teuthology.orchestra.run.smithi098.stdout:242 write 0x138fef3 thru 0x139427a (0x4388 bytes) 2024-09-16T13:16:57.884 INFO:teuthology.orchestra.run.smithi098.stdout:243 trunc from 0xa739260 to 0x47da68b 2024-09-16T13:16:57.893 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:57.888+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:57.897 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:16:57.919 INFO:teuthology.orchestra.run.smithi098.stdout:244 write 0x847beb8 thru 0x847cf34 (0x107d bytes) 2024-09-16T13:16:57.924 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:16:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:16:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:16:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:16:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:16:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:16:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:16:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:16:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:16:57.931 INFO:teuthology.orchestra.run.smithi098.stdout:245 trunc from 0x847cf35 to 0x53ff741 2024-09-16T13:16:57.955 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:16:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:16:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:16:57.956 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:57.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:16:57.975 INFO:teuthology.orchestra.run.smithi098.stdout:246 writesame 0xea104e1 thru 0xea19e24 data_size 0x22(0x9944 bytes) 2024-09-16T13:16:57.996 INFO:teuthology.orchestra.run.smithi098.stdout:247 trunc from 0xea19e25 to 0x1af12c4 2024-09-16T13:16:57.996 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:57.992+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:58.013 INFO:teuthology.orchestra.run.smithi098.stdout:248 read 0xa2154e thru 0xa2d100 (0xbbb3 bytes) 2024-09-16T13:16:58.029 INFO:teuthology.orchestra.run.smithi098.stdout:249 write 0x401ba92 thru 0x4023558 (0x7ac7 bytes) 2024-09-16T13:16:58.035 INFO:teuthology.orchestra.run.smithi098.stdout:250 read 0x3a90bb6 thru 0x3a98bd7 (0x8022 bytes) 2024-09-16T13:16:58.035 INFO:teuthology.orchestra.run.smithi098.stdout:251 read 0x20f8c07 thru 0x20fab39 (0x1f33 bytes) 2024-09-16T13:16:58.044 INFO:teuthology.orchestra.run.smithi098.stdout:252 writesame 0x4666fd1 thru 0x466c7f3 data_size 0x9(0x5823 bytes) 2024-09-16T13:16:58.067 INFO:teuthology.orchestra.run.smithi098.stdout:253 write 0xb990f22 thru 0xb99f79e (0xe87d bytes) 2024-09-16T13:16:58.084 INFO:teuthology.orchestra.run.smithi098.stdout:255 trunc from 0xb99f79f to 0x5ed4d33 2024-09-16T13:16:58.095 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:58.088+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:58.100 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:16:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:16:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:16:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:16:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:647 3.2 3:46e985a0:::rbd_data.112de98401a1.000000000000002e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.100 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.816569+0000", 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.275219347, 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035756490000000002, 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 647 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.816569+0000", 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.816569+0000", 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.816570+0000", 2024-09-16T13:16:58.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.816573+0000", 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5319999999999998e-06 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.816581+0000", 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2220000000000002e-06 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.816586+0000", 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9099999999999996e-06 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.816646+0000", 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9901999999999997e-05 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.816738+0000", 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1391999999999995e-05 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.816866+0000", 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012801699999999999 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.819203+0000", 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002337184 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.820015+0000", 2024-09-16T13:16:58.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000812213 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.820102+0000", 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6644999999999995e-05 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.820145+0000", 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2936999999999999e-05 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:670 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.850932+0000", 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.24085614599999999, 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002526978, 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 670 2024-09-16T13:16:58.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.850932+0000", 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.850932+0000", 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.850935+0000", 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3810000000000002e-06 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.850947+0000", 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2114e-05 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.850973+0000", 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6083000000000001e-05 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.850982+0000", 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7040000000000006e-06 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.851080+0000", 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7756000000000004e-05 2024-09-16T13:16:58.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.851257+0000", 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017678799999999999 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.851768+0000", 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00051087699999999997 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.853129+0000", 2024-09-16T13:16:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013613270000000001 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.853372+0000", 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024269299999999999 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.853422+0000", 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0145999999999997e-05 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.853459+0000", 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7109e-05 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:679 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.859908+0000", 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.2318808, 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027547829999999998, 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.113 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 679 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.859908+0000", 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.859908+0000", 2024-09-16T13:16:58.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.859909+0000", 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9379999999999999e-06 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.859919+0000", 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6670000000000008e-06 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.859943+0000", 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4116000000000001e-05 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.859954+0000", 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0528e-05 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.860045+0000", 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1308000000000001e-05 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.860224+0000", 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017925499999999999 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.861064+0000", 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00084000399999999999 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.862418+0000", 2024-09-16T13:16:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013532030000000001 2024-09-16T13:16:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.862611+0000", 2024-09-16T13:16:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019326900000000001 2024-09-16T13:16:58.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.862641+0000", 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0000000000000001e-05 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.862662+0000", 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1495e-05 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:682 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.881462+0000", 2024-09-16T13:16:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.21032642700000001, 2024-09-16T13:16:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024728530000000001, 2024-09-16T13:16:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 682 2024-09-16T13:16:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.881462+0000", 2024-09-16T13:16:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.881462+0000", 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.881464+0000", 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3379999999999999e-06 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.881476+0000", 2024-09-16T13:16:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1793999999999999e-05 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.881499+0000", 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2544e-05 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.881507+0000", 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7050000000000005e-06 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.881631+0000", 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012326399999999999 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.881764+0000", 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000133428 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.882220+0000", 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045571199999999999 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.883551+0000", 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013308720000000001 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.883813+0000", 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000262735 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.883868+0000", 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4761e-05 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.883935+0000", 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6699999999999995e-05 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:696 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.905447+0000", 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.186341691, 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029933759999999998, 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 696 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.905447+0000", 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.905447+0000", 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.905449+0000", 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8090000000000001e-06 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.905466+0000", 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6475999999999999e-05 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.905498+0000", 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2053e-05 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.905510+0000", 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2016e-05 2024-09-16T13:16:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.905625+0000", 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011485999999999999 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.905789+0000", 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000164587 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.906262+0000", 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000473024 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.907630+0000", 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013675180000000001 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908260+0000", 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00063042100000000004 2024-09-16T13:16:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908367+0000", 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106822 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908440+0000", 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2789999999999999e-05 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:697 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.908597+0000", 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.18319112000000001, 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029095660000000001, 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 697 2024-09-16T13:16:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908597+0000", 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908597+0000", 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908598+0000", 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1310000000000001e-06 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908604+0000", 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908621+0000", 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7478000000000001e-05 2024-09-16T13:16:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908631+0000", 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5990000000000008e-06 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908713+0000", 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1563000000000005e-05 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.908879+0000", 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000166229 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.909264+0000", 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00038501099999999999 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.910352+0000", 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010885580000000001 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.911402+0000", 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010495560000000001 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.911460+0000", 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7911999999999997e-05 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.911507+0000", 2024-09-16T13:16:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6947000000000003e-05 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:705 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.915882+0000", 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.175906601, 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002491563, 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 705 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.915882+0000", 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.915882+0000", 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.915883+0000", 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.915888+0000", 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.916e-06 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.915902+0000", 2024-09-16T13:16:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4285e-05 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.915908+0000", 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3890000000000002e-06 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.915969+0000", 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1192000000000006e-05 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.916084+0000", 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114147 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.916521+0000", 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043750400000000002 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.917667+0000", 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011462670000000001 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.918249+0000", 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058142600000000001 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.918313+0000", 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3925999999999997e-05 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.918373+0000", 2024-09-16T13:16:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0652999999999997e-05 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:707 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.927802+0000", 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.16398610799999999, 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002837558, 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 707 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.927802+0000", 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.927802+0000", 2024-09-16T13:16:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.927805+0000", 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.357e-06 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.927817+0000", 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2323e-05 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.927842+0000", 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4663000000000001e-05 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.927852+0000", 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0010000000000001e-05 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.927920+0000", 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8689999999999995e-05 2024-09-16T13:16:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.928051+0000", 2024-09-16T13:16:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000130569 2024-09-16T13:16:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.928489+0000", 2024-09-16T13:16:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043856400000000001 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.930242+0000", 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001752366 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.930534+0000", 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029250000000000001 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.930577+0000", 2024-09-16T13:16:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2218999999999997e-05 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.930640+0000", 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3297000000000001e-05 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:710 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.935802+0000", 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.15598621600000001, 2024-09-16T13:16:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025210279999999998, 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 710 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.935802+0000", 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.935802+0000", 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.935804+0000", 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.415e-06 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.935809+0000", 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4389999999999999e-06 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.935823+0000", 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3594000000000001e-05 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.935832+0000", 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7310000000000005e-06 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.935899+0000", 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6469000000000003e-05 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.936024+0000", 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012551500000000001 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.936347+0000", 2024-09-16T13:16:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00032316400000000003 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.937170+0000", 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082204900000000004 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.938239+0000", 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010691559999999999 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.938273+0000", 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.464e-05 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.938323+0000", 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9855999999999998e-05 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:719 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.957371+0000", 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.13441753200000001, 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028347170000000001, 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 719 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.957371+0000", 2024-09-16T13:16:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.957371+0000", 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.957373+0000", 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.542e-06 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.957386+0000", 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2753e-05 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.957411+0000", 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4905999999999998e-05 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.957423+0000", 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1639e-05 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.957548+0000", 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012524199999999999 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.957768+0000", 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022022599999999999 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.958396+0000", 2024-09-16T13:16:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00062761900000000005 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.959852+0000", 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001455767 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960105+0000", 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025392299999999997 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960159+0000", 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960206+0000", 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6579999999999998e-05 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:720 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.960418+0000", 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.131370179, 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070090029999999998, 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 720 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960418+0000", 2024-09-16T13:16:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960418+0000", 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960420+0000", 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7239999999999999e-06 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960428+0000", 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5590000000000006e-06 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960440+0000", 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.188e-05 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960445+0000", 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8149999999999996e-06 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960562+0000", 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116747 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960676+0000", 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114431 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.960942+0000", 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0002656 2024-09-16T13:16:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.961789+0000", 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00084687900000000003 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.967357+0000", 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055676969999999999 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.967397+0000", 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9993999999999997e-05 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.967427+0000", 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0676999999999998e-05 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:728 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.992789+0000", 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.098999749999999997, 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026554479999999999, 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 728 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.992789+0000", 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.992789+0000", 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.992791+0000", 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2809999999999998e-06 2024-09-16T13:16:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.992803+0000", 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1657e-05 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.992827+0000", 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4898000000000001e-05 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.992837+0000", 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1630000000000002e-06 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.992904+0000", 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7058999999999996e-05 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.992986+0000", 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2044000000000003e-05 2024-09-16T13:16:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.993257+0000", 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027175000000000002 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.994305+0000", 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001047753 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.995364+0000", 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010590619999999999 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.995400+0000", 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6074000000000001e-05 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.995444+0000", 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3707e-05 2024-09-16T13:16:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:731 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.999945+0000", 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.091843818999999993, 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026280299999999999, 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 731 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.999945+0000", 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.999945+0000", 2024-09-16T13:16:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.999945+0000", 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0599999999999999e-07 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.999951+0000", 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8660000000000002e-06 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.999961+0000", 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6439999999999995e-06 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.999966+0000", 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5800000000000002e-06 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.000028+0000", 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2602999999999997e-05 2024-09-16T13:16:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.000127+0000", 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9137999999999998e-05 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.000386+0000", 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025856799999999998 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.000953+0000", 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056758399999999999 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.002461+0000", 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001508037 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.002528+0000", 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6120999999999999e-05 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.002573+0000", 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4982999999999999e-05 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:741 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.015324+0000", 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.076464751999999997, 2024-09-16T13:16:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024149950000000001, 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 741 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.015324+0000", 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.015324+0000", 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.015326+0000", 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2299999999999998e-06 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.015339+0000", 2024-09-16T13:16:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3073e-05 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.015364+0000", 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5554e-05 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.015376+0000", 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1083e-05 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.015476+0000", 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000100257 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.015610+0000", 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013371699999999999 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.015969+0000", 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000359327 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.017265+0000", 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001296607 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.017636+0000", 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037038499999999999 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.017692+0000", 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5778e-05 2024-09-16T13:16:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.017739+0000", 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6984000000000003e-05 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:742 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.017907+0000", 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.073881815000000003, 2024-09-16T13:16:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025364630000000001, 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 742 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.017907+0000", 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.017907+0000", 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.017908+0000", 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0890000000000001e-06 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.017913+0000", 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9869999999999999e-06 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.017925+0000", 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.22e-05 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.017931+0000", 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7810000000000002e-06 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.018009+0000", 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8780000000000001e-05 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.018133+0000", 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123788 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.018390+0000", 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025633900000000001 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.019381+0000", 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00099134599999999994 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.020337+0000", 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00095659300000000005 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.020397+0000", 2024-09-16T13:16:58.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9098999999999998e-05 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.020443+0000", 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461000000000001e-05 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:750 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.025581+0000", 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.066207337000000005, 2024-09-16T13:16:58.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027244140000000001, 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 750 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.025581+0000", 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.025581+0000", 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.025583+0000", 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7600000000000001e-06 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.025592+0000", 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2450000000000004e-06 2024-09-16T13:16:58.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.025619+0000", 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7248000000000001e-05 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.025630+0000", 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.026e-05 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.025727+0000", 2024-09-16T13:16:58.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7798000000000001e-05 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.025875+0000", 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014755599999999999 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.026494+0000", 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00061907800000000001 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.027974+0000", 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014797619999999999 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.028210+0000", 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023600699999999999 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.028266+0000", 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.605e-05 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.028305+0000", 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9650000000000002e-05 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:753 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.035418+0000", 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.056370300999999998, 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002376464, 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 753 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.035418+0000", 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.035418+0000", 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.035419+0000", 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0270000000000001e-06 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.035424+0000", 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4469999999999999e-06 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.035435+0000", 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1847e-05 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.035442+0000", 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7170000000000002e-06 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.035503+0000", 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0463e-05 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.035593+0000", 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0527999999999998e-05 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.035800+0000", 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000207262 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.036643+0000", 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00084296700000000002 2024-09-16T13:16:58.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.037693+0000", 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001049269 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.037746+0000", 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3748000000000001e-05 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.037795+0000", 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8189000000000002e-05 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:762 3.2 3:4640f636:::rbd_header.112de98401a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.056510+0000", 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.035278811, 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026348389999999999, 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.206 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 762 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.056510+0000", 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.056510+0000", 2024-09-16T13:16:58.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.056512+0000", 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0650000000000001e-06 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.056521+0000", 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8830000000000001e-06 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.056546+0000", 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4930999999999999e-05 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.056557+0000", 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.028e-05 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.056612+0000", 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5714999999999998e-05 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.056685+0000", 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2886999999999995e-05 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.056854+0000", 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016870700000000001 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.057552+0000", 2024-09-16T13:16:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000698396 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.058890+0000", 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001337481 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.059032+0000", 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014195399999999999 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.059144+0000", 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011254 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:770 3.8 3:12e71eb6:::rbd_object_map.112de98401a1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.064540+0000", 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.02724822, 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029068219999999999, 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.212 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 770 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.064540+0000", 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.064540+0000", 2024-09-16T13:16:58.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.064542+0000", 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7400000000000001e-06 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.064553+0000", 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209000000000001e-05 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.064577+0000", 2024-09-16T13:16:58.214 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4193999999999999e-05 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.064587+0000", 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0016e-05 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.064656+0000", 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8585999999999997e-05 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.064757+0000", 2024-09-16T13:16:58.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000100758 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.065173+0000", 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041630900000000001 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.066630+0000", 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014568809999999999 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067249+0000", 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00061888399999999999 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067340+0000", 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1178e-05 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067447+0000", 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107067 2024-09-16T13:16:58.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:771 3.2 3:46e985a0:::rbd_data.112de98401a1.000000000000002e:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1642274~59517] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.067576+0000", 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.024212621, 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003201569, 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:16:58.218 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 771 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067576+0000", 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067576+0000", 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067578+0000", 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.03e-06 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067677+0000", 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8944999999999995e-05 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067703+0000", 2024-09-16T13:16:58.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5851999999999999e-05 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067713+0000", 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0842999999999999e-05 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067796+0000", 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2688999999999994e-05 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.067883+0000", 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6973999999999998e-05 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.068024+0000", 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014100500000000001 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.069323+0000", 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00129879 2024-09-16T13:16:58.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.070570+0000", 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001247352 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.070668+0000", 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7831000000000006e-05 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.070777+0000", 2024-09-16T13:16:58.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109258 2024-09-16T13:16:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:16:58.224 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:16:58.224 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:16:58.225 INFO:teuthology.orchestra.run.smithi098.stdout:258 punch from 0x1e08006 to 0x1e08b81, (0xb7b bytes) 2024-09-16T13:16:58.225 INFO:teuthology.orchestra.run.smithi098.stdout:259 read 0x6b2797 thru 0x6b36ff (0xf69 bytes) 2024-09-16T13:16:58.225 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:16:58.227 INFO:teuthology.orchestra.run.smithi098.stdout:261 write 0xe1b2312 thru 0xe1b75e9 (0x52d8 bytes) 2024-09-16T13:16:58.228 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.228 INFO:teuthology.orchestra.run.smithi098.stdout:264 writesame 0x78a726 thru 0x79648f data_size 0xd(0xbd6a bytes) 2024-09-16T13:16:58.228 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.228 INFO:teuthology.orchestra.run.smithi098.stdout:268 writesame 0x6ca0acf thru 0x6ca5d3c data_size 0x3(0x526e bytes) 2024-09-16T13:16:58.228 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.228 INFO:teuthology.orchestra.run.smithi098.stdout:270 write 0xd5e286c thru 0xd5e7134 (0x48c9 bytes) 2024-09-16T13:16:58.228 INFO:teuthology.orchestra.run.smithi098.stdout:271 write 0xe11e0c9 thru 0xe12d2ce (0xf206 bytes) 2024-09-16T13:16:58.228 INFO:teuthology.orchestra.run.smithi098.stdout:272 trunc from 0xe1b75ea to 0x75d7d0b 2024-09-16T13:16:58.228 INFO:teuthology.orchestra.run.smithi098.stdout:274 write 0x525709f thru 0x5263441 (0xc3a3 bytes) 2024-09-16T13:16:58.228 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:58.220+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:58.231 INFO:teuthology.orchestra.run.smithi098.stdout:275 write 0xa886075 thru 0xa890c19 (0xaba5 bytes) 2024-09-16T13:16:58.238 INFO:teuthology.orchestra.run.smithi098.stdout:277 punch from 0x1810676 to 0x1818143, (0x7acd bytes) 2024-09-16T13:16:58.239 INFO:teuthology.orchestra.run.smithi098.stdout:278 write 0x94ffde4 thru 0x950298e (0x2bab bytes) 2024-09-16T13:16:58.246 INFO:teuthology.orchestra.run.smithi098.stdout:279 read 0x1b4b382 thru 0x1b5a25c (0xeedb bytes) 2024-09-16T13:16:58.246 INFO:teuthology.orchestra.run.smithi098.stdout:280 write 0x6db9a79 thru 0x6dc3da8 (0xa330 bytes) 2024-09-16T13:16:58.250 INFO:teuthology.orchestra.run.smithi098.stdout:281 punch from 0xe9d81 to 0xf2108, (0x8387 bytes) 2024-09-16T13:16:58.250 INFO:teuthology.orchestra.run.smithi098.stdout:282 read 0xa0f8c19 thru 0xa10372a (0xab12 bytes) 2024-09-16T13:16:58.255 INFO:teuthology.orchestra.run.smithi098.stdout:283 trunc from 0xa890c1a to 0xcac9a32 2024-09-16T13:16:58.267 INFO:teuthology.orchestra.run.smithi098.stdout:284 write 0xca85e8 thru 0xcaefcd (0x69e6 bytes) 2024-09-16T13:16:58.286 INFO:teuthology.orchestra.run.smithi098.stdout:285 trunc from 0xcac9a32 to 0x77a1c01 2024-09-16T13:16:58.308 INFO:teuthology.orchestra.run.smithi098.stdout:286 read 0x7445846 thru 0x7448297 (0x2a52 bytes) 2024-09-16T13:16:58.308 INFO:teuthology.orchestra.run.smithi098.stdout:288 read 0x5858987 thru 0x5863317 (0xa991 bytes) 2024-09-16T13:16:58.308 INFO:teuthology.orchestra.run.smithi098.stdout:290 read 0x2a540a0 thru 0x2a624f7 (0xe458 bytes) 2024-09-16T13:16:58.326 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:58.320+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:16:58.328 INFO:teuthology.orchestra.run.smithi098.stdout:291 writesame 0xbd92f83 thru 0xbd9d8b3 data_size 0x1(0xa931 bytes) 2024-09-16T13:16:58.329 INFO:teuthology.orchestra.run.smithi098.stdout:292 trunc from 0xbd9d8b4 to 0xb9e79be 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:871 3.1a 3.ac0e573a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.335232+0000", 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.1761000000000005e-05, 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013023700000000001, 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.339 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 871 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335232+0000", 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335232+0000", 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335234+0000", 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3599999999999999e-06 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335248+0000", 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3662999999999999e-05 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335280+0000", 2024-09-16T13:16:58.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1888999999999999e-05 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335293+0000", 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2794e-05 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 1 2024-09-16T13:16:58.342 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:58.343 INFO:teuthology.orchestra.run.smithi098.stdout:293 read 0x50c0da9 thru 0x50cbebe (0xb116 bytes) 2024-09-16T13:16:58.343 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.343 INFO:teuthology.orchestra.run.smithi098.stdout:295 read 0x2208a7d thru 0x2208fcb (0x54f bytes) 2024-09-16T13:16:58.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:16:58.362 INFO:teuthology.orchestra.run.smithi098.stdout:296 trunc from 0xb9e79be to 0x9e557 2024-09-16T13:16:58.384 INFO:teuthology.orchestra.run.smithi098.stdout:297 punch from 0x15b49 to 0x1f212, (0x96c9 bytes) 2024-09-16T13:16:58.409 INFO:teuthology.orchestra.run.smithi098.stdout:299 trunc from 0x9e557 to 0xd7957ee 2024-09-16T13:16:58.421 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.422 INFO:teuthology.orchestra.run.smithi098.stdout:302 write 0x96f69f4 thru 0x96f8b0f (0x211c bytes) 2024-09-16T13:16:58.427 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:58.420+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:16:58.428 INFO:teuthology.orchestra.run.smithi098.stdout:303 write 0x330f035 thru 0x331dc8a (0xec56 bytes) 2024-09-16T13:16:58.435 INFO:teuthology.orchestra.run.smithi098.stdout:304 writesame 0x7fb5758 thru 0x7fb8fe7 data_size 0x28(0x3890 bytes) 2024-09-16T13:16:58.441 INFO:teuthology.orchestra.run.smithi098.stdout:305 write 0xc57b944 thru 0xc588a3a (0xd0f7 bytes) 2024-09-16T13:16:58.445 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:58.445 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:58.446 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:16:58.446 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:16:58.446 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:58.448 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:16:58.468 INFO:teuthology.orchestra.run.smithi098.stdout:309 trunc from 0xd7957ee to 0x37ad1fc 2024-09-16T13:16:58.486 INFO:teuthology.orchestra.run.smithi098.stdout:310 read 0x14136d2 thru 0x141d32c (0x9c5b bytes) 2024-09-16T13:16:58.495 INFO:teuthology.orchestra.run.smithi098.stdout:311 trunc from 0x37ad1fc to 0x7ee7dec 2024-09-16T13:16:58.505 INFO:teuthology.orchestra.run.smithi098.stdout:312 punch from 0x49df36a to 0x49e689a, (0x7530 bytes) 2024-09-16T13:16:58.505 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.505 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.505 INFO:teuthology.orchestra.run.smithi098.stdout:316 read 0x40bffbf thru 0x40cddf6 (0xde38 bytes) 2024-09-16T13:16:58.523 INFO:teuthology.orchestra.run.smithi098.stdout:317 writesame 0xb33675c thru 0xb345b61 data_size 0x5(0xf406 bytes) 2024-09-16T13:16:58.528 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:16:58.522+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:16:58.535 INFO:teuthology.orchestra.run.smithi098.stdout:318 trunc from 0xb345b62 to 0x502b457 2024-09-16T13:16:58.547 INFO:teuthology.orchestra.run.smithi098.stdout:320 write 0xfc5290 thru 0xfc6b51 (0x18c2 bytes) 2024-09-16T13:16:58.559 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:58.559 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:7 3.1a 3:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:53.287525+0000", 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.2650851110000003, 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033858310000000002, 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.560 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.287525+0000", 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.287525+0000", 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.287528+0000", 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.033e-06 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.287548+0000", 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9987e-05 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.287582+0000", 2024-09-16T13:16:58.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4604999999999998e-05 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.287596+0000", 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3067e-05 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.287723+0000", 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012752400000000001 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.288064+0000", 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034050200000000001 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.288454+0000", 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039000500000000002 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.290017+0000", 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015634679999999999 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.290732+0000", 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00071527899999999998 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.290807+0000", 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4611999999999996e-05 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:53.290911+0000", 2024-09-16T13:16:58.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103749 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:9 3.1a 3:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:54.299570+0000", 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.2530404409999996, 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070841239999999998, 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:58.566 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299570+0000", 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299570+0000", 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299573+0000", 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3170000000000002e-06 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299587+0000", 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4389999999999999e-05 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299624+0000", 2024-09-16T13:16:58.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6944999999999999e-05 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299645+0000", 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0947000000000001e-05 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299793+0000", 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014779800000000001 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299805+0000", 2024-09-16T13:16:58.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2248e-05 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.303287+0000", 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034814350000000002 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.303461+0000", 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000174234 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.303611+0000", 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000150605 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.305822+0000", 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022106589999999998 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.306554+0000", 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073181600000000004 2024-09-16T13:16:58.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.306608+0000", 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3946999999999997e-05 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.306654+0000", 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5782999999999999e-05 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:10 3.1c e26/23 3:3831950c:::rbd_directory:head v 26'1, mlcod=0'0)", 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:54.308681+0000", 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.2439286770000004, 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001142182, 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.308681+0000", 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.308681+0000", 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.308682+0000", 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3399999999999997e-07 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.308692+0000", 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9939999999999995e-06 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.308711+0000", 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8479999999999999e-05 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.308721+0000", 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0795000000000001e-05 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.308803+0000", 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1590999999999999e-05 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.308846+0000", 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2478999999999998e-05 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.309759+0000", 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00091296599999999997 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.309823+0000", 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4943000000000005e-05 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:677 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:57.858661+0000", 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.69394920100000002, 2024-09-16T13:16:58.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00079480499999999999, 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 677 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.858661+0000", 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.858661+0000", 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.858663+0000", 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.323e-06 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.858675+0000", 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2103000000000001e-05 2024-09-16T13:16:58.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.858707+0000", 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1639e-05 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.858719+0000", 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2500000000000001e-05 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.858878+0000", 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015820299999999999 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.859113+0000", 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235868 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.859456+0000", 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034216900000000002 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.581 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.582 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.582 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:680 3.1b e27/23 3:dda6f6d6:::rbd_data.112de98401a1.0000000000000029:head v 27'1, mlcod=0'0)", 2024-09-16T13:16:58.582 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:57.863898+0000", 2024-09-16T13:16:58.582 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.68871226799999996, 2024-09-16T13:16:58.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002093831, 2024-09-16T13:16:58.582 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.582 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.863898+0000", 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.863898+0000", 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.863900+0000", 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.136e-06 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.863949+0000", 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8683000000000001e-05 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.863981+0000", 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2941999999999997e-05 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.863993+0000", 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.154e-05 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.864088+0000", 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5321000000000004e-05 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.864128+0000", 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9743000000000002e-05 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.865899+0000", 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017707770000000001 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.865992+0000", 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2689000000000007e-05 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:685 3.1b e27/23 3:dda6f6d6:::rbd_data.112de98401a1.0000000000000029:head v 27'2, mlcod=0'0)", 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:57.885095+0000", 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.66751529700000001, 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001893348, 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.885095+0000", 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.885095+0000", 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.885097+0000", 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.885112+0000", 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5486e-05 2024-09-16T13:16:58.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.885144+0000", 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2143e-05 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.885156+0000", 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1562999999999999e-05 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.885339+0000", 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000182766 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.885387+0000", 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8602e-05 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.886880+0000", 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001493294 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.886988+0000", 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107567 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:706 3.1d e27/23 3:bc8f37a2:::rbd_data.112de98401a1.0000000000000021:head v 27'10, mlcod=27'8)", 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:57.919664+0000", 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.63294583299999996, 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019778230000000001, 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919664+0000", 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919664+0000", 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919666+0000", 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.68e-06 2024-09-16T13:16:58.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919685+0000", 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8893999999999999e-05 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919717+0000", 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2271999999999999e-05 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919730+0000", 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3346999999999999e-05 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919849+0000", 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118897 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919909+0000", 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0089e-05 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.921500+0000", 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001590307 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.921642+0000", 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142337 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:708 3.1d e27/23 3:bc8f37a2:::rbd_data.112de98401a1.0000000000000021:head v 27'11, mlcod=27'9)", 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:57.931854+0000", 2024-09-16T13:16:58.595 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.62075549100000005, 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001412005, 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.931854+0000", 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.931854+0000", 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.931856+0000", 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.863e-06 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.931870+0000", 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.399e-05 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.931902+0000", 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1279000000000003e-05 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.931915+0000", 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3397000000000001e-05 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.932020+0000", 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104671 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.932062+0000", 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1845999999999999e-05 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.933181+0000", 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011193609999999999 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.933266+0000", 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5598000000000003e-05 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:798 3.9 3:940847c8:::rbd_data.112de98401a1.000000000000001b:head [set-alloc-hint object_size 4194304 write_size 4194304,write 658127~21102] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.149479+0000", 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.40313048000000001, 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034890619999999998, 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 798 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.600 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.149479+0000", 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.149479+0000", 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.149482+0000", 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3769999999999999e-06 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.149648+0000", 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016602300000000001 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.149680+0000", 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2459000000000002e-05 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.149693+0000", 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2186e-05 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.149849+0000", 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015678400000000001 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.150231+0000", 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038147399999999997 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.150713+0000", 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00048192999999999999 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.152819+0000", 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021065599999999999 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.152841+0000", 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2226e-05 2024-09-16T13:16:58.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.152915+0000", 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3931999999999996e-05 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.152969+0000", 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3111e-05 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:829 3.11 e27/23 3:8a48d229:::rbd_data.112de98401a1.0000000000000025:head v 27'1, mlcod=0'0)", 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.240835+0000", 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.31177532299999999, 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001242438, 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.240835+0000", 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.240835+0000", 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.240837+0000", 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5129999999999999e-06 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.240875+0000", 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.807e-05 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.240921+0000", 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5841999999999999e-05 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.240964+0000", 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3192000000000003e-05 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.241049+0000", 2024-09-16T13:16:58.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5119999999999998e-05 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.241071+0000", 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1746000000000002e-05 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.242024+0000", 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00095311699999999996 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.242077+0000", 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2837999999999998e-05 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:830 3.9 3:940847c8:::rbd_data.112de98401a1.000000000000001b:head [write 1809017~41776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.243480+0000", 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.30912958299999999, 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032988409999999998, 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.610 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 830 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.243480+0000", 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.243480+0000", 2024-09-16T13:16:58.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.243487+0000", 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6290000000000004e-06 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.243553+0000", 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6161999999999994e-05 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.243571+0000", 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7774999999999998e-05 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.243577+0000", 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1650000000000003e-06 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.243680+0000", 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102859 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.243846+0000", 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000165773 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.244063+0000", 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021770900000000001 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.245691+0000", 2024-09-16T13:16:58.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001627248 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.246501+0000", 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00081076300000000004 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.246660+0000", 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015805699999999999 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.246779+0000", 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119701 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:843 3.11 e27/23 3:8a48d229:::rbd_data.112de98401a1.0000000000000025:head v 27'2, mlcod=0'0)", 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.287693+0000", 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.26491705700000001, 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001384861, 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.287693+0000", 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.287693+0000", 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.287696+0000", 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800000000000002e-06 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.287712+0000", 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5713e-05 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.287748+0000", 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5914999999999997e-05 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.287760+0000", 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2490000000000001e-05 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.287835+0000", 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5121000000000001e-05 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.287863+0000", 2024-09-16T13:16:58.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7708999999999998e-05 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.289028+0000", 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001165361 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.289078+0000", 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9472000000000003e-05 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.620 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:862 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.323310+0000", 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.22929970999999999, 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00077457899999999996, 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 862 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.621 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.323310+0000", 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.323310+0000", 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.323313+0000", 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4760000000000001e-06 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.323325+0000", 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2743e-05 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.323357+0000", 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.184e-05 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.323373+0000", 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5676e-05 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.323503+0000", 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130514 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.323760+0000", 2024-09-16T13:16:58.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025688399999999998 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.324085+0000", 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032444599999999999 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:871 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.335232+0000", 2024-09-16T13:16:58.625 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.21737793899999999, 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00075571000000000004, 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 871 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335232+0000", 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335232+0000", 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335234+0000", 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3599999999999999e-06 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335248+0000", 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3662999999999999e-05 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335280+0000", 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1888999999999999e-05 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335293+0000", 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2794e-05 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335417+0000", 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012463700000000001 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335679+0000", 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026135399999999999 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.335988+0000", 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030901300000000002 2024-09-16T13:16:58.629 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:879 3.3 3:c7f7a3a4:::rbd_data.112de98401a1.0000000000000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.362894+0000", 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.18971597600000001, 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030576370000000002, 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:58.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 879 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362894+0000", 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362894+0000", 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362897+0000", 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.937e-06 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362917+0000", 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9843e-05 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362949+0000", 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2511000000000001e-05 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362967+0000", 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7756000000000001e-05 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363063+0000", 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6180000000000004e-05 2024-09-16T13:16:58.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363166+0000", 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102429 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363296+0000", 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013080399999999999 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.365315+0000", 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020188810000000001 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.365676+0000", 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036050600000000002 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.365817+0000", 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014164000000000001 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.365952+0000", 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013415000000000001 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:882 3.9 3:940847c8:::rbd_data.112de98401a1.000000000000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.363009+0000", 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.18960049800000001, 2024-09-16T13:16:58.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046613410000000003, 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 882 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363009+0000", 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363009+0000", 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363011+0000", 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3799999999999999e-06 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363018+0000", 2024-09-16T13:16:58.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8630000000000002e-06 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363034+0000", 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6279000000000001e-05 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363045+0000", 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1216999999999999e-05 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363162+0000", 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117166 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363340+0000", 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000177782 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363591+0000", 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025106 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.365435+0000", 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018437010000000001 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.367578+0000", 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021431670000000001 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.367634+0000", 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5747e-05 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.367671+0000", 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6979e-05 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:904 3.11 e27/23 3:8a48d229:::rbd_data.112de98401a1.0000000000000025:head v 27'3, mlcod=27'1)", 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.422936+0000", 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.12967384100000001, 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001927984, 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.642 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.422936+0000", 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.422936+0000", 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.422938+0000", 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.747e-06 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.422964+0000", 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5925999999999999e-05 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.422994+0000", 2024-09-16T13:16:58.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0273e-05 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.423005+0000", 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1386000000000001e-05 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.423138+0000", 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013205700000000001 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.423210+0000", 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2051999999999994e-05 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.424799+0000", 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001589381 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.424864+0000", 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5161999999999997e-05 2024-09-16T13:16:58.646 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:910 3.13 e27/23 3:cb88fb25:::rbd_data.112de98401a1.0000000000000031:head v 27'11, mlcod=27'9)", 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.442335+0000", 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.11027469299999999, 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023529309999999999, 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.442335+0000", 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.442335+0000", 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.442337+0000", 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.59e-06 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.442405+0000", 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8355999999999994e-05 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.442436+0000", 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0539e-05 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.442453+0000", 2024-09-16T13:16:58.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7016000000000001e-05 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.442578+0000", 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012500099999999999 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.442637+0000", 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9194999999999998e-05 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.444609+0000", 2024-09-16T13:16:58.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019721890000000001 2024-09-16T13:16:58.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.444688+0000", 2024-09-16T13:16:58.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9045e-05 2024-09-16T13:16:58.651 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.651 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:914 3.13 e27/23 3:cb88fb25:::rbd_data.112de98401a1.0000000000000031:head v 27'12, mlcod=27'10)", 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.468515+0000", 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.084094855999999996, 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001809887, 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468515+0000", 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468515+0000", 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468517+0000", 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2699999999999999e-06 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468539+0000", 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1149e-05 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468585+0000", 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6802999999999999e-05 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468605+0000", 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9358999999999999e-05 2024-09-16T13:16:58.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468720+0000", 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114888 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468778+0000", 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8687999999999998e-05 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.470293+0000", 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001514678 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.470325+0000", 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2051999999999997e-05 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:913 3.11 e27/23 3:8a48d229:::rbd_data.112de98401a1.0000000000000025:head v 27'4, mlcod=27'2)", 2024-09-16T13:16:58.656 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.468645+0000", 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.083965165999999994, 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001134361, 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468645+0000", 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:58.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468645+0000", 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468646+0000", 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8000000000000004e-07 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468656+0000", 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0604e-05 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:58.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468674+0000", 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7476e-05 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468686+0000", 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2053e-05 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468817+0000", 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131605 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.468845+0000", 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7296000000000001e-05 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.469727+0000", 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00088233799999999998 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.469779+0000", 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2108999999999999e-05 2024-09-16T13:16:58.660 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.661 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:58.661 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:58.661 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:58.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:16:58.664 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:58.658+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:58.664 INFO:teuthology.orchestra.run.smithi098.stdout:321 write 0xa25b4eb thru 0xa266b94 (0xb6aa bytes) 2024-09-16T13:16:58.664 INFO:teuthology.orchestra.run.smithi098.stdout:322 write 0xeb31650 thru 0xeb33d64 (0x2715 bytes) 2024-09-16T13:16:58.664 INFO:teuthology.orchestra.run.smithi098.stdout:323 trunc from 0xeb33d65 to 0x3da35e9 2024-09-16T13:16:58.664 INFO:teuthology.orchestra.run.smithi098.stdout:324 write 0x30abfad thru 0x30b4d3d (0x8d91 bytes) 2024-09-16T13:16:58.664 INFO:teuthology.orchestra.run.smithi098.stdout:325 writesame 0x33a6ac9 thru 0x33ad0d4 data_size 0x2a(0x660c bytes) 2024-09-16T13:16:58.664 INFO:teuthology.orchestra.run.smithi098.stdout:326 read 0x1a37fc4 thru 0x1a4281c (0xa859 bytes) 2024-09-16T13:16:58.664 INFO:teuthology.orchestra.run.smithi098.stdout:327 trunc from 0x3da35e9 to 0x2eb870c 2024-09-16T13:16:58.678 INFO:teuthology.orchestra.run.smithi098.stdout:328 punch from 0x5e43b1 to 0x5e6134, (0x1d83 bytes) 2024-09-16T13:16:58.707 INFO:teuthology.orchestra.run.smithi098.stdout:329 writesame 0xb9d56fe thru 0xb9d64a7 data_size 0x21(0xdaa bytes) 2024-09-16T13:16:58.723 INFO:teuthology.orchestra.run.smithi098.stdout:330 writesame 0xccaa020 thru 0xccb2474 data_size 0x1(0x8455 bytes) 2024-09-16T13:16:58.723 INFO:teuthology.orchestra.run.smithi098.stdout:331 read 0x89827c9 thru 0x8989378 (0x6bb0 bytes) 2024-09-16T13:16:58.723 INFO:teuthology.orchestra.run.smithi098.stdout:332 writesame 0xb4cacd7 thru 0xb4cd161 data_size 0x1(0x248b bytes) 2024-09-16T13:16:58.740 INFO:teuthology.orchestra.run.smithi098.stdout:334 write 0xa1fdfed thru 0xa20502d (0x7041 bytes) 2024-09-16T13:16:58.748 INFO:teuthology.orchestra.run.smithi098.stdout:335 writesame 0xc18497 thru 0xc282e9 data_size 0x7(0xfe53 bytes) 2024-09-16T13:16:58.763 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:16:58.756+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:16:58.764 INFO:teuthology.orchestra.run.smithi098.stdout:336 write 0x5b68e28 thru 0x5b7431c (0xb4f5 bytes) 2024-09-16T13:16:58.770 INFO:teuthology.orchestra.run.smithi098.stdout:338 write 0xcc8a56d thru 0xcc8b687 (0x111b bytes) 2024-09-16T13:16:58.776 INFO:teuthology.orchestra.run.smithi098.stdout:339 write 0x24ed5d9 thru 0x24f3df5 (0x681d bytes) 2024-09-16T13:16:58.780 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:58.780 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:58.780 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:16:58.780 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:58.783 INFO:teuthology.orchestra.run.smithi098.stdout:340 read 0x878eef8 thru 0x87952bd (0x63c6 bytes) 2024-09-16T13:16:58.783 INFO:teuthology.orchestra.run.smithi098.stdout:341 punch from 0xc95fd4a to 0xc960d2e, (0xfe4 bytes) 2024-09-16T13:16:58.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:16:58.798 INFO:teuthology.orchestra.run.smithi098.stdout:342 trunc from 0xccb2475 to 0x4d29c4e 2024-09-16T13:16:58.836 INFO:teuthology.orchestra.run.smithi098.stdout:343 write 0x61f3b03 thru 0x6202ab4 (0xefb2 bytes) 2024-09-16T13:16:58.843 INFO:teuthology.orchestra.run.smithi098.stdout:344 punch from 0x4c773f9 to 0x4c81977, (0xa57e bytes) 2024-09-16T13:16:58.852 INFO:teuthology.orchestra.run.smithi098.stdout:345 trunc from 0x6202ab5 to 0x9d86c55 2024-09-16T13:16:58.864 INFO:teuthology.orchestra.run.smithi098.stdout:346 read 0x81c2ed5 thru 0x81cc7d3 (0x98ff bytes) 2024-09-16T13:16:58.864 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.864 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.865 INFO:teuthology.orchestra.run.smithi098.stdout:349 write 0x3a7c4d5 thru 0x3a89385 (0xceb1 bytes) 2024-09-16T13:16:58.865 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:58.854+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:58.883 INFO:teuthology.orchestra.run.smithi098.stdout:350 trunc from 0x9d86c55 to 0x3be668b 2024-09-16T13:16:58.887 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:58.887 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:58.887 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:16:58.887 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:16:58.887 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:58.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:16:58.907 INFO:teuthology.orchestra.run.smithi098.stdout:351 trunc from 0x3be668b to 0x4b13e34 2024-09-16T13:16:58.918 INFO:teuthology.orchestra.run.smithi098.stdout:353 read 0x2aeb877 thru 0x2af9ce3 (0xe46d bytes) 2024-09-16T13:16:58.918 INFO:teuthology.orchestra.run.smithi098.stdout:355 read 0x41875e4 thru 0x4187dbd (0x7da bytes) 2024-09-16T13:16:58.919 INFO:teuthology.orchestra.run.smithi098.stdout:357 punch from 0x3bef671 to 0x3bfaf5e, (0xb8ed bytes) 2024-09-16T13:16:58.919 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.933 INFO:teuthology.orchestra.run.smithi098.stdout:360 trunc from 0x4b13e34 to 0xbe32d4b 2024-09-16T13:16:58.943 INFO:teuthology.orchestra.run.smithi098.stdout:362 read 0x8078c78 thru 0x8086371 (0xd6fa bytes) 2024-09-16T13:16:58.949 INFO:teuthology.orchestra.run.smithi098.stdout:363 trunc from 0xbe32d4b to 0x87865f5 2024-09-16T13:16:58.960 INFO:teuthology.orchestra.run.smithi098.stdout:364 read 0x15bcdb8 thru 0x15c3e9b (0x70e4 bytes) 2024-09-16T13:16:58.960 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.960 INFO:teuthology.orchestra.run.smithi098.stdout:366 punch from 0x28133cb to 0x28172ed, (0x3f22 bytes) 2024-09-16T13:16:58.960 INFO:teuthology.orchestra.run.smithi098.stdout:367 write 0x404dbfa thru 0x4054c21 (0x7028 bytes) 2024-09-16T13:16:58.965 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:16:58.954+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:16:58.966 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.966 INFO:teuthology.orchestra.run.smithi098.stdout:370 writesame 0x617884e thru 0x617be21 data_size 0x35(0x35d4 bytes) 2024-09-16T13:16:58.973 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:58.973 INFO:teuthology.orchestra.run.smithi098.stdout:373 read 0x1bc4025 thru 0x1bd06cd (0xc6a9 bytes) 2024-09-16T13:16:58.973 INFO:teuthology.orchestra.run.smithi098.stdout:374 read 0x73b2361 thru 0x73b9a79 (0x7719 bytes) 2024-09-16T13:16:58.974 INFO:teuthology.orchestra.run.smithi098.stdout:375 trunc from 0x87865f5 to 0x8d5fbb0 2024-09-16T13:16:58.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:16:58.984 INFO:teuthology.orchestra.run.smithi098.stdout:376 write 0x2bb074d thru 0x2bb2053 (0x1907 bytes) 2024-09-16T13:16:58.989 INFO:teuthology.orchestra.run.smithi098.stdout:378 read 0xb9893f thru 0xba4e96 (0xc558 bytes) 2024-09-16T13:16:58.996 INFO:teuthology.orchestra.run.smithi098.stdout:379 trunc from 0x8d5fbb0 to 0x5ae724b 2024-09-16T13:16:58.998 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:16:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:16:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4281.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:42.824475+0000", 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.166477739000001, 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057003050000000001, 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4281", 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/2946615432", 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.824475+0000", 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.824475+0000", 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.824485+0000", 2024-09-16T13:16:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0159999999999999e-05 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.824502+0000", 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6217000000000001e-05 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.824522+0000", 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0509e-05 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.824535+0000", 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2773e-05 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.824668+0000", 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013289000000000001 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.824694+0000", 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.605e-05 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.828165+0000", 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003470866 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.828436+0000", 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027138800000000001 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:16:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.828571+0000", 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000134671 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.829608+0000", 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001037336 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.830083+0000", 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000474539 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.830123+0000", 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0621999999999998e-05 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:42.830176+0000", 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2284000000000003e-05 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4281.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:43.828090+0000", 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.162862978, 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089561039999999995, 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4281", 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/2946615432", 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 5 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.828090+0000", 2024-09-16T13:16:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.828090+0000", 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.828094+0000", 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8299999999999998e-06 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.828122+0000", 2024-09-16T13:16:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7702e-05 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.828171+0000", 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9412000000000001e-05 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.828191+0000", 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9570000000000001e-05 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.828339+0000", 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014888299999999999 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.828363+0000", 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3706999999999999e-05 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.833431+0000", 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050677839999999997 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.833654+0000", 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022269799999999999 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.833880+0000", 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022614799999999999 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.835898+0000", 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020183050000000002 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.836796+0000", 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00089819699999999999 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.836941+0000", 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014436700000000001 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:43.837046+0000", 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010550100000000001 2024-09-16T13:16:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4281.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:44.835362+0000", 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.155591018999999, 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0092252110000000005, 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4281", 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/2946615432", 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.835362+0000", 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.835362+0000", 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.835365+0000", 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3330000000000001e-06 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.835378+0000", 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2903e-05 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.835417+0000", 2024-09-16T13:16:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9091999999999998e-05 2024-09-16T13:16:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.835432+0000", 2024-09-16T13:16:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4406999999999999e-05 2024-09-16T13:16:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.835568+0000", 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013676399999999999 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.835585+0000", 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6283e-05 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.840930+0000", 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0053454440000000004 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.841141+0000", 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021051199999999999 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.841546+0000", 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000405294 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.843357+0000", 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001810636 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.844364+0000", 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010077910000000001 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.844473+0000", 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108195 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:44.844587+0000", 2024-09-16T13:16:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011455699999999999 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:706 3.1d 3:bc8f37a2:::rbd_data.112de98401a1.0000000000000021:head [set-alloc-hint object_size 4194304 write_size 4194304,write 507576~4221] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:57.918844+0000", 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.072108692, 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034469090000000002, 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 706 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.918844+0000", 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.918844+0000", 2024-09-16T13:16:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.918847+0000", 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.633e-06 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.918870+0000", 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3402999999999999e-05 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.918914+0000", 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3432e-05 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.918931+0000", 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7407000000000001e-05 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919069+0000", 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013736 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919217+0000", 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148913 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.919457+0000", 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000239638 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.921380+0000", 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019231070000000001 2024-09-16T13:16:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.922127+0000", 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00074730300000000005 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.922218+0000", 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0142000000000005e-05 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.922291+0000", 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3571000000000004e-05 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:720 3.2 e27/23 3:4640f636:::rbd_header.112de98401a1:head v 27'63, mlcod=27'61)", 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:57.961424+0000", 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.0295293539999999, 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0059932559999999998, 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.961424+0000", 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.961424+0000", 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.961425+0000", 2024-09-16T13:16:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.683e-06 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.961459+0000", 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3445999999999998e-05 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.961491+0000", 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1751999999999997e-05 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.961502+0000", 2024-09-16T13:16:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1094000000000001e-05 2024-09-16T13:16:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.961599+0000", 2024-09-16T13:16:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6923999999999996e-05 2024-09-16T13:16:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.961639+0000", 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0929000000000001e-05 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.967303+0000", 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056632410000000003 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:57.967417+0000", 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114187 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:751 3.c 3:361f3f00:::rbd_data.112de98401a1.0000000000000010:head [set-alloc-hint object_size 4194304 write_size 4194304,write 113298~31431] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.028954+0000", 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.96199895000000002, 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029049620000000001, 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 751 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.028954+0000", 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.028954+0000", 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.028957+0000", 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6929999999999999e-06 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.029026+0000", 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9624999999999998e-05 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.029064+0000", 2024-09-16T13:16:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7735999999999999e-05 2024-09-16T13:16:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.029077+0000", 2024-09-16T13:16:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2619000000000001e-05 2024-09-16T13:16:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.029214+0000", 2024-09-16T13:16:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013723699999999999 2024-09-16T13:16:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.029362+0000", 2024-09-16T13:16:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014845700000000001 2024-09-16T13:16:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.029619+0000", 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000256221 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.031649+0000", 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020303180000000001 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.031685+0000", 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5623000000000001e-05 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.031778+0000", 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3640999999999993e-05 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.031859+0000", 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0791999999999995e-05 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:795 3.16 3:6d48d4ea:::rbd_data.112de98401a1.0000000000000038:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1778450~21208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.129697+0000", 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.86125593499999997, 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003264884, 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 795 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.129697+0000", 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.129697+0000", 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.129699+0000", 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3829999999999999e-06 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.129731+0000", 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1930000000000001e-05 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.129769+0000", 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7731999999999997e-05 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.129786+0000", 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7240000000000001e-05 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.129960+0000", 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017375299999999999 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.130105+0000", 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144926 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.130352+0000", 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024654199999999999 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.131976+0000", 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001624454 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.132808+0000", 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00083208800000000003 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.132897+0000", 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9327999999999996e-05 2024-09-16T13:16:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.132962+0000", 2024-09-16T13:16:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4508e-05 2024-09-16T13:16:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:796 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:head [write 3712806~48490] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.136798+0000", 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.85415507599999996, 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031927560000000002, 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 796 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.136798+0000", 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.136798+0000", 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.136800+0000", 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3499999999999999e-06 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.136862+0000", 2024-09-16T13:16:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2129000000000001e-05 2024-09-16T13:16:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.136897+0000", 2024-09-16T13:16:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5049000000000003e-05 2024-09-16T13:16:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.136911+0000", 2024-09-16T13:16:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4015999999999999e-05 2024-09-16T13:16:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.137060+0000", 2024-09-16T13:16:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014896700000000001 2024-09-16T13:16:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.137301+0000", 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024052500000000001 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.137713+0000", 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041212099999999999 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.139762+0000", 2024-09-16T13:16:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020487439999999999 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.139826+0000", 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4401999999999998e-05 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.139916+0000", 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9877999999999996e-05 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.139991+0000", 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4574999999999997e-05 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:801 3.16 3:6d48d4ea:::rbd_data.112de98401a1.0000000000000038:head [write 1171657~61958] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.161760+0000", 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.82919337800000004, 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031675700000000002, 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 801 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.161760+0000", 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.161760+0000", 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.161762+0000", 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.04e-06 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.161870+0000", 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108578 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.161901+0000", 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0574999999999998e-05 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.161913+0000", 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1827e-05 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.162067+0000", 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015463299999999999 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.162223+0000", 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155797 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.162453+0000", 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000230251 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.164116+0000", 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001662567 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.164772+0000", 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00065605800000000005 2024-09-16T13:16:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.164861+0000", 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8887999999999993e-05 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.164927+0000", 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6356e-05 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:804 3.16 3:6d48d4ea:::rbd_data.112de98401a1.0000000000000038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.183883+0000", 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.80706950700000002, 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006074131, 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 804 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.183883+0000", 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.183883+0000", 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.183886+0000", 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:16:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.183896+0000", 2024-09-16T13:16:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0793e-05 2024-09-16T13:16:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.183933+0000", 2024-09-16T13:16:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6732e-05 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.183950+0000", 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6956000000000002e-05 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.184122+0000", 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017201299999999999 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.184367+0000", 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024513200000000001 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.184747+0000", 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037956099999999999 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.186539+0000", 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001791967 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.189763+0000", 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003223868 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.189856+0000", 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3281000000000004e-05 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.189958+0000", 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101718 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:803 3.d e27/23 3:b6e80fb1:::rbd_data.112de98401a1.0000000000000035:head v 27'16, mlcod=27'14)", 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.184927+0000", 2024-09-16T13:16:59.062 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.80602582, 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045259389999999997, 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.184927+0000", 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.184927+0000", 2024-09-16T13:16:59.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.184928+0000", 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.319e-06 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.184943+0000", 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4399999999999999e-05 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.184968+0000", 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4689999999999999e-05 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.184982+0000", 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3976e-05 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.185208+0000", 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022668499999999999 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.185328+0000", 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012014399999999999 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.189374+0000", 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040452530000000004 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.189453+0000", 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9472000000000001e-05 2024-09-16T13:16:59.066 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:875 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.362906+0000", 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.62804669999999996, 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032484250000000001, 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.067 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 875 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362906+0000", 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362906+0000", 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362909+0000", 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.757e-06 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362925+0000", 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5526999999999999e-05 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362961+0000", 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6418999999999998e-05 2024-09-16T13:16:59.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.362976+0000", 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4525999999999999e-05 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363071+0000", 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5446999999999996e-05 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363166+0000", 2024-09-16T13:16:59.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.467e-05 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363298+0000", 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000132063 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.365226+0000", 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001928007 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.365952+0000", 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00072590700000000003 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.366080+0000", 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128534 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.366155+0000", 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4567999999999995e-05 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.072 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:878 3.5 3:a35423a9:::rbd_data.112de98401a1.0000000000000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.363018+0000", 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.62793461800000006, 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046772200000000002, 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 878 2024-09-16T13:16:59.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363018+0000", 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363018+0000", 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363020+0000", 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3859999999999999e-06 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363026+0000", 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5869999999999998e-06 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363045+0000", 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8862e-05 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363058+0000", 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3074000000000001e-05 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363248+0000", 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018958899999999999 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363440+0000", 2024-09-16T13:16:59.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019246500000000001 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363799+0000", 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035866199999999999 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.365654+0000", 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001855258 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.367564+0000", 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019097999999999999 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.367641+0000", 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6959999999999995e-05 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.367696+0000", 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4577000000000003e-05 2024-09-16T13:16:59.078 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:880 3.c 3:361f3f00:::rbd_data.112de98401a1.0000000000000010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.363088+0000", 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.62786481699999996, 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0052698830000000004, 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.079 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 880 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363088+0000", 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363088+0000", 2024-09-16T13:16:59.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363090+0000", 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.333e-06 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363096+0000", 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2639999999999997e-06 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363114+0000", 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8196e-05 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363125+0000", 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1106e-05 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.364338+0000", 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012127329999999999 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.364664+0000", 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000326533 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.364791+0000", 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126515 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.367659+0000", 2024-09-16T13:16:59.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028685289999999999 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.368207+0000", 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00054791099999999999 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.368290+0000", 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2997999999999994e-05 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.368358+0000", 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7764999999999999e-05 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:876 3.d e27/23 3:b0dc6c85:::rbd_data.112de98401a1.0000000000000002:head v 27'19, mlcod=27'17)", 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.363852+0000", 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.62710136800000005, 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036959850000000002, 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363852+0000", 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363852+0000", 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363853+0000", 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3170000000000001e-06 2024-09-16T13:16:59.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363882+0000", 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8827999999999999e-05 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363899+0000", 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7498000000000001e-05 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.363916+0000", 2024-09-16T13:16:59.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7065999999999999e-05 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.364006+0000", 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9543000000000006e-05 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.364038+0000", 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2276000000000001e-05 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.367420+0000", 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003381837 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.367548+0000", 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012762 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:964 3.1c 3:3d928fe3:::rbd_data.112de98401a1.0000000000000028:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2471147~46762] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.089 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.575604+0000", 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.41534933200000002, 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028122720000000002, 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 964 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.575604+0000", 2024-09-16T13:16:59.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.575604+0000", 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.575606+0000", 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8700000000000001e-06 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.575662+0000", 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6765999999999998e-05 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.575682+0000", 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0183e-05 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.575689+0000", 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.575746+0000", 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6669000000000002e-05 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.575868+0000", 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122528 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.575996+0000", 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012795300000000001 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.577358+0000", 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001361365 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.578326+0000", 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00096860000000000002 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.578369+0000", 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2330999999999998e-05 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.578416+0000", 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7358999999999998e-05 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:977 3.1c 3:3d928fe3:::rbd_data.112de98401a1.0000000000000028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.629535+0000", 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.361417612, 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028722660000000001, 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.095 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 977 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.629535+0000", 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.629535+0000", 2024-09-16T13:16:59.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.629537+0000", 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7600000000000001e-06 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.629547+0000", 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9969999999999993e-06 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.629577+0000", 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0287999999999999e-05 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.629591+0000", 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3264999999999999e-05 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.629689+0000", 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7918000000000007e-05 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.629803+0000", 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011414899999999999 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.629943+0000", 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000140548 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.631341+0000", 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013979140000000001 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.632211+0000", 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00086992099999999995 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.632325+0000", 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011400099999999999 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.632408+0000", 2024-09-16T13:16:59.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2504999999999997e-05 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1024 3.1c 3:3d928fe3:::rbd_data.112de98401a1.0000000000000028:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2088941~28737] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.741633+0000", 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.24931951099999999, 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032794069999999998, 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.101 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1024 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.741633+0000", 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.741633+0000", 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.741637+0000", 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1599999999999998e-06 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.741704+0000", 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7707999999999996e-05 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.741742+0000", 2024-09-16T13:16:59.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8039e-05 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.741756+0000", 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3657e-05 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.741911+0000", 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015472700000000001 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.742154+0000", 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000242814 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.742616+0000", 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046238699999999998 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.744438+0000", 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018223989999999999 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.744723+0000", 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028479699999999999 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.744819+0000", 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5577999999999999e-05 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.744913+0000", 2024-09-16T13:16:59.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4141000000000005e-05 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1029 3.5 3:a6efa855:::rbd_data.112de98401a1.0000000000000033:head [set-alloc-hint object_size 4194304 write_size 4194304,write 566637~4379] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.770474+0000", 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.22047871699999999, 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030004839999999999, 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1029 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.770474+0000", 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.770474+0000", 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.770477+0000", 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2400000000000002e-06 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.770489+0000", 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2972999999999999e-05 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.770512+0000", 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2469000000000001e-05 2024-09-16T13:16:59.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.770520+0000", 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8150000000000006e-06 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.770621+0000", 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010128000000000001 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.770778+0000", 2024-09-16T13:16:59.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015709900000000001 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.771094+0000", 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000315877 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.772150+0000", 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010562359999999999 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.773349+0000", 2024-09-16T13:16:59.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011985710000000001 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.773411+0000", 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.19e-05 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.773475+0000", 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4023999999999996e-05 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1036 3.5 3:a6efa855:::rbd_data.112de98401a1.0000000000000033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.800575+0000", 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.190377819, 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029146839999999999, 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1036 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.113 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800575+0000", 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800575+0000", 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800576+0000", 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9299999999999998e-07 2024-09-16T13:16:59.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800579+0000", 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7110000000000001e-06 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800586+0000", 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7669999999999998e-06 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800592+0000", 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4369999999999998e-06 2024-09-16T13:16:59.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800645+0000", 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3554999999999998e-05 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800747+0000", 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102124 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:16:59.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800916+0000", 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016830600000000001 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.802433+0000", 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001517443 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.803185+0000", 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00075189099999999997 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.803342+0000", 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015734700000000001 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.803490+0000", 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014751 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.118 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:59.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:16:59.121 INFO:teuthology.orchestra.run.smithi098.stdout:380 read 0x3856a42 thru 0x3863eae (0xd46d bytes) 2024-09-16T13:16:59.121 INFO:teuthology.orchestra.run.smithi098.stdout:384 punch from 0x1847332 to 0x184ebd2, (0x78a0 bytes) 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:386 trunc from 0x5ae724b to 0xb9692f4 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:389 writesame 0x550be30 thru 0x551a87a data_size 0x1(0xea4b bytes) 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:390 read 0x253449d thru 0x25421c7 (0xdd2b bytes) 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:391 read 0x2247e9c thru 0x224be3e (0x3fa3 bytes) 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:392 write 0x8eff92b thru 0x8f028fb (0x2fd1 bytes) 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:393 read 0xb4070bf thru 0xb41332b (0xc26d bytes) 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:395 read 0x6ead96d thru 0x6eb5111 (0x77a5 bytes) 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:397 read 0x8375c7 thru 0x838cfe (0x1738 bytes) 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:398 write 0x1de9f85 thru 0x1df5778 (0xb7f4 bytes) 2024-09-16T13:16:59.122 INFO:teuthology.orchestra.run.smithi098.stdout:399 writesame 0xa90f085 thru 0xa91af3b data_size 0x1(0xbeb7 bytes) 2024-09-16T13:16:59.123 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:59.112+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:59.141 INFO:teuthology.orchestra.run.smithi098.stdout:truncating to largest ever: 0xe84ef34 2024-09-16T13:16:59.145 INFO:teuthology.orchestra.run.smithi098.stdout:401 trunc from 0xb9692f4 to 0xe84ef34 2024-09-16T13:16:59.157 INFO:teuthology.orchestra.run.smithi098.stdout:403 punch from 0x8d22496 to 0x8d2ad3a, (0x88a4 bytes) 2024-09-16T13:16:59.157 INFO:teuthology.orchestra.run.smithi098.stdout:404 punch from 0x2c4afdb to 0x2c57fe8, (0xd00d bytes) 2024-09-16T13:16:59.157 INFO:teuthology.orchestra.run.smithi098.stdout:405 write 0x4e3d503 thru 0x4e3dd1c (0x81a bytes) 2024-09-16T13:16:59.164 INFO:teuthology.orchestra.run.smithi098.stdout:406 read 0x7724a4c thru 0x772e335 (0x98ea bytes) 2024-09-16T13:16:59.164 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.164 INFO:teuthology.orchestra.run.smithi098.stdout:409 write 0xc95a753 thru 0xc95bb91 (0x143f bytes) 2024-09-16T13:16:59.189 INFO:teuthology.orchestra.run.smithi098.stdout:411 trunc from 0xe84ef34 to 0x3fa5466 2024-09-16T13:16:59.216 INFO:teuthology.orchestra.run.smithi098.stdout:412 trunc from 0x3fa5466 to 0x7478603 2024-09-16T13:16:59.220 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:59.216+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:16:59.227 INFO:teuthology.orchestra.run.smithi098.stdout:413 write 0xcd7bf1 thru 0xcd923e (0x164e bytes) 2024-09-16T13:16:59.241 INFO:teuthology.orchestra.run.smithi098.stdout:415 writesame 0x84ff4fc thru 0x8502b32 data_size 0x1(0x3637 bytes) 2024-09-16T13:16:59.265 INFO:teuthology.orchestra.run.smithi098.stdout:416 write 0xed2f65b thru 0xed359bd (0x6363 bytes) 2024-09-16T13:16:59.270 INFO:teuthology.orchestra.run.smithi098.stdout:419 writesame 0x2eef597 thru 0x2efb110 data_size 0x39(0xbb7a bytes) 2024-09-16T13:16:59.275 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:59.275 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:59.275 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:16:59.275 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:59.277 INFO:teuthology.orchestra.run.smithi098.stdout:420 read 0x8a1dd5a thru 0x8a27672 (0x9919 bytes) 2024-09-16T13:16:59.277 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.277 INFO:teuthology.orchestra.run.smithi098.stdout:423 write 0x5b1513d thru 0x5b1741a (0x22de bytes) 2024-09-16T13:16:59.283 INFO:teuthology.orchestra.run.smithi098.stdout:426 punch from 0xb2bf864 to 0xb2c3ae9, (0x4285 bytes) 2024-09-16T13:16:59.283 INFO:teuthology.orchestra.run.smithi098.stdout:427 writesame 0x2e8ca3 thru 0x2f6caf data_size 0x3(0xe00d bytes) 2024-09-16T13:16:59.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:16:59.300 INFO:teuthology.orchestra.run.smithi098.stdout:428 punch from 0x7f2ea21 to 0x7f32919, (0x3ef8 bytes) 2024-09-16T13:16:59.300 INFO:teuthology.orchestra.run.smithi098.stdout:429 read 0x9ae336b thru 0x9ae62cd (0x2f63 bytes) 2024-09-16T13:16:59.300 INFO:teuthology.orchestra.run.smithi098.stdout:430 read 0xb871856 thru 0xb87a391 (0x8b3c bytes) 2024-09-16T13:16:59.300 INFO:teuthology.orchestra.run.smithi098.stdout:431 read 0x3dc2d4e thru 0x3dce181 (0xb434 bytes) 2024-09-16T13:16:59.300 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.300 INFO:teuthology.orchestra.run.smithi098.stdout:433 write 0x6bee72e thru 0x6bfcc9c (0xe56f bytes) 2024-09-16T13:16:59.306 INFO:teuthology.orchestra.run.smithi098.stdout:434 write 0x1003f7a thru 0x10094ab (0x5532 bytes) 2024-09-16T13:16:59.312 INFO:teuthology.orchestra.run.smithi098.stdout:435 read 0xcaa5496 thru 0xcaacf7b (0x7ae6 bytes) 2024-09-16T13:16:59.312 INFO:teuthology.orchestra.run.smithi098.stdout:436 writesame 0xf92110 thru 0xf97e75 data_size 0xf(0x5d66 bytes) 2024-09-16T13:16:59.317 INFO:teuthology.orchestra.run.smithi098.stdout:437 write 0x87a1349 thru 0x87a6c82 (0x593a bytes) 2024-09-16T13:16:59.322 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:16:59.318+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:16:59.323 INFO:teuthology.orchestra.run.smithi098.stdout:442 write 0x8c29d76 thru 0x8c38f88 (0xf213 bytes) 2024-09-16T13:16:59.338 INFO:teuthology.orchestra.run.smithi098.stdout:443 trunc from 0xed359be to 0xad197ba 2024-09-16T13:16:59.360 INFO:teuthology.orchestra.run.smithi098.stdout:444 writesame 0xa2e9571 thru 0xa2ea12b data_size 0x7(0xbbb bytes) 2024-09-16T13:16:59.367 INFO:teuthology.orchestra.run.smithi098.stdout:445 punch from 0x52f7f3f to 0x52fad81, (0x2e42 bytes) 2024-09-16T13:16:59.381 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:59.381 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:59.381 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:16:59.381 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:16:59.382 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:59.385 INFO:teuthology.orchestra.run.smithi098.stdout:446 writesame 0xcb88e48 thru 0xcb93272 data_size 0x1(0xa42b bytes) 2024-09-16T13:16:59.385 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.385 INFO:teuthology.orchestra.run.smithi098.stdout:448 write 0x83b36e1 thru 0x83c0149 (0xca69 bytes) 2024-09-16T13:16:59.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:16:59.391 INFO:teuthology.orchestra.run.smithi098.stdout:449 write 0x941e2d6 thru 0x9427cdd (0x9a08 bytes) 2024-09-16T13:16:59.398 INFO:teuthology.orchestra.run.smithi098.stdout:450 punch from 0xc46bedb to 0xc472052, (0x6177 bytes) 2024-09-16T13:16:59.398 INFO:teuthology.orchestra.run.smithi098.stdout:451 write 0x2d0b689 thru 0x2d143ec (0x8d64 bytes) 2024-09-16T13:16:59.401 INFO:teuthology.orchestra.run.smithi098.stdout:452 writesame 0x4ef194e thru 0x4f009fc data_size 0x1(0xf0af bytes) 2024-09-16T13:16:59.422 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:59.416+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:16:59.461 INFO:teuthology.orchestra.run.smithi098.stdout:453 write 0x59abd4f thru 0x59ac03a (0x2ec bytes) 2024-09-16T13:16:59.468 INFO:teuthology.orchestra.run.smithi098.stdout:454 write 0x82a6504 thru 0x82acd3f (0x683c bytes) 2024-09-16T13:16:59.495 INFO:teuthology.orchestra.run.smithi098.stdout:456 trunc from 0xcb93273 to 0x15a5daa 2024-09-16T13:16:59.523 INFO:teuthology.orchestra.run.smithi098.stdout:457 read 0x10717fb thru 0x107a0f1 (0x88f7 bytes) 2024-09-16T13:16:59.523 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:16:59.516+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:16:59.539 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:59.539 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:16:59.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:16:59.539 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1022 3.10 3:0c4f3e94:::rbd_data.112de98401a1.000000000000002d:head [set-alloc-hint object_size 4194304 write_size 4194304,write 830679~9355] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.732415+0000", 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.80303290599999999, 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040114139999999996, 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1022 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.540 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732415+0000", 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732415+0000", 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732418+0000", 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2770000000000001e-06 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732942+0000", 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052370100000000001 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732986+0000", 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4079000000000003e-05 2024-09-16T13:16:59.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733002+0000", 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5716999999999999e-05 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733137+0000", 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013541700000000001 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733389+0000", 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025132399999999998 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733884+0000", 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049490199999999997 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736097+0000", 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002213338 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736272+0000", 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017514900000000001 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736363+0000", 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1467999999999999e-05 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736427+0000", 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3041999999999997e-05 2024-09-16T13:16:59.545 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.4397.0:1022 3.10 e27/23 ondisk, result = 0)", 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.736061+0000", 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.79938714700000002, 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038179799999999998, 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736061+0000", 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736061+0000", 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736063+0000", 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6070000000000002e-06 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736077+0000", 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3212e-05 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736105+0000", 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8212999999999999e-05 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736134+0000", 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8875999999999999e-05 2024-09-16T13:16:59.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736238+0000", 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010402499999999999 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736256+0000", 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7866000000000001e-05 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736443+0000", 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186999 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1035 3.10 3:0c4f3e94:::rbd_data.112de98401a1.000000000000002d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.800590+0000", 2024-09-16T13:16:59.550 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.73485814800000004, 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025099089999999998, 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1035 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800590+0000", 2024-09-16T13:16:59.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800590+0000", 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800593+0000", 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5269999999999999e-06 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800610+0000", 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6396e-05 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800667+0000", 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6836000000000003e-05 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800693+0000", 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6125000000000001e-05 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.800854+0000", 2024-09-16T13:16:59.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000161364 2024-09-16T13:16:59.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.801141+0000", 2024-09-16T13:16:59.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028693300000000001 2024-09-16T13:16:59.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.801434+0000", 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029330699999999998 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.802860+0000", 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014252850000000001 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.803018+0000", 2024-09-16T13:16:59.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000158195 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.803054+0000", 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6497e-05 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.803100+0000", 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5444000000000001e-05 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.4397.0:1035 3.10 e27/23 ondisk, result = 0)", 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.802910+0000", 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.73253749899999998, 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019387000000000001, 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.802910+0000", 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.802910+0000", 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.802912+0000", 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0920000000000001e-06 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.802918+0000", 2024-09-16T13:16:59.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.032e-06 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.802934+0000", 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6812000000000001e-05 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.802961+0000", 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6757000000000001e-05 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.803007+0000", 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5952000000000003e-05 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.803013+0000", 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8699999999999997e-06 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.803104+0000", 2024-09-16T13:16:59.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1354999999999995e-05 2024-09-16T13:16:59.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.561 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:59.563 INFO:teuthology.orchestra.run.smithi098.stdout:460 write 0x54c6b04 thru 0x54cedcc (0x82c9 bytes) 2024-09-16T13:16:59.563 INFO:teuthology.orchestra.run.smithi098.stdout:462 punch from 0x44e6bf6 to 0x44ea56d, (0x3977 bytes) 2024-09-16T13:16:59.563 INFO:teuthology.orchestra.run.smithi098.stdout:463 read 0x3b489ed thru 0x3b54d0b (0xc31f bytes) 2024-09-16T13:16:59.563 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.563 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.563 INFO:teuthology.orchestra.run.smithi098.stdout:467 read 0x398bbd1 thru 0x39969b7 (0xade7 bytes) 2024-09-16T13:16:59.563 INFO:teuthology.orchestra.run.smithi098.stdout:468 writesame 0x1a3d560 thru 0x1a3e550 data_size 0xb(0xff1 bytes) 2024-09-16T13:16:59.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:16:59.572 INFO:teuthology.orchestra.run.smithi098.stdout:469 write 0x9fc594b thru 0x9fcce31 (0x74e7 bytes) 2024-09-16T13:16:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:472 write 0x19e9242 thru 0x19f274f (0x950e bytes) 2024-09-16T13:16:59.582 INFO:teuthology.orchestra.run.smithi098.stdout:473 read 0x703bbe0 thru 0x703cbd6 (0xff7 bytes) 2024-09-16T13:16:59.582 INFO:teuthology.orchestra.run.smithi098.stdout:474 read 0x98ea078 thru 0x98ecbe0 (0x2b69 bytes) 2024-09-16T13:16:59.597 INFO:teuthology.orchestra.run.smithi098.stdout:475 write 0xbbd7bf5 thru 0xbbdd771 (0x5b7d bytes) 2024-09-16T13:16:59.603 INFO:teuthology.orchestra.run.smithi098.stdout:476 read 0x8db670b thru 0x8dc262c (0xbf22 bytes) 2024-09-16T13:16:59.603 INFO:teuthology.orchestra.run.smithi098.stdout:477 punch from 0x575bdff to 0x575f7f7, (0x39f8 bytes) 2024-09-16T13:16:59.625 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:59.620+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:59.626 INFO:teuthology.orchestra.run.smithi098.stdout:478 trunc from 0xbbdd772 to 0x13d6045 2024-09-16T13:16:59.666 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:59.666 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:59.666 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:16:59.666 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:59.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:16:59.685 INFO:teuthology.orchestra.run.smithi098.stdout:479 write 0xd50591c thru 0xd510bde (0xb2c3 bytes) 2024-09-16T13:16:59.690 INFO:teuthology.orchestra.run.smithi098.stdout:481 writesame 0x795950 thru 0x79ac81 data_size 0x17(0x5332 bytes) 2024-09-16T13:16:59.697 INFO:teuthology.orchestra.run.smithi098.stdout:482 write 0x96353d thru 0x972aee (0xf5b2 bytes) 2024-09-16T13:16:59.703 INFO:teuthology.orchestra.run.smithi098.stdout:483 read 0x2088eb1 thru 0x208ef54 (0x60a4 bytes) 2024-09-16T13:16:59.703 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.703 INFO:teuthology.orchestra.run.smithi098.stdout:485 write 0xc337534 thru 0xc340445 (0x8f12 bytes) 2024-09-16T13:16:59.723 INFO:teuthology.orchestra.run.smithi098.stdout:486 trunc from 0xd510bdf to 0x7044d64 2024-09-16T13:16:59.725 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:16:59.720+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:16:59.743 INFO:teuthology.orchestra.run.smithi098.stdout:487 punch from 0x4ed8fbd to 0x4ee0c81, (0x7cc4 bytes) 2024-09-16T13:16:59.743 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.752 INFO:teuthology.orchestra.run.smithi098.stdout:489 trunc from 0x7044d64 to 0x20d503a 2024-09-16T13:16:59.764 INFO:teuthology.orchestra.run.smithi098.stdout:490 trunc from 0x20d503a to 0x2b713cd 2024-09-16T13:16:59.774 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:59.774 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:16:59.774 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:16:59.774 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:16:59.774 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:16:59.774 INFO:teuthology.orchestra.run.smithi098.stdout:493 read 0x95e78c thru 0x96023f (0x1ab4 bytes) 2024-09-16T13:16:59.776 INFO:teuthology.orchestra.run.smithi098.stdout:494 trunc from 0x2b713cd to 0x2e99d68 2024-09-16T13:16:59.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:16:59.786 INFO:teuthology.orchestra.run.smithi098.stdout:truncating to largest ever: 0xee00c85 2024-09-16T13:16:59.809 INFO:teuthology.orchestra.run.smithi098.stdout:495 trunc from 0x2e99d68 to 0xee00c85 2024-09-16T13:16:59.820 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:16:59.820 INFO:teuthology.orchestra.run.smithi098.stdout:498 writesame 0x97d8fac thru 0x97d9c6f data_size 0x4(0xcc4 bytes) 2024-09-16T13:16:59.826 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:16:59.820+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:16:59.827 INFO:teuthology.orchestra.run.smithi098.stdout:500 writesame 0x9bd0f4b thru 0x9bd3ccf data_size 0x1(0x2d85 bytes) 2024-09-16T13:16:59.845 INFO:teuthology.orchestra.run.smithi098.stdout:501 read 0x63e2b77 thru 0x63ef113 (0xc59d bytes) 2024-09-16T13:16:59.845 INFO:teuthology.orchestra.run.smithi098.stdout:502 write 0x4b778ce thru 0x4b7b55f (0x3c92 bytes) 2024-09-16T13:16:59.852 INFO:teuthology.orchestra.run.smithi098.stdout:503 writesame 0xafa76af thru 0xafaef73 data_size 0x1(0x78c5 bytes) 2024-09-16T13:16:59.885 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:16:59.885 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:16:59.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.024918+0000", 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.852235681, 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0098686339999999994, 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 44 2024-09-16T13:16:59.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024918+0000", 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024918+0000", 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024919+0000", 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.093e-06 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024997+0000", 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8595000000000002e-05 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025009+0000", 2024-09-16T13:16:59.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2006e-05 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025016+0000", 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999999e-06 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.031767+0000", 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067515229999999997 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.031892+0000", 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012455100000000001 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.032088+0000", 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000196152 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034632+0000", 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025435430000000001 2024-09-16T13:16:59.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034658+0000", 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6784999999999999e-05 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034734+0000", 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5482000000000007e-05 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034786+0000", 2024-09-16T13:16:59.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2173999999999999e-05 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025030+0000", 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.852123094, 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011426475, 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.892 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025030+0000", 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025030+0000", 2024-09-16T13:16:59.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025031+0000", 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0100000000000001e-06 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025037+0000", 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4600000000000002e-06 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025046+0000", 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2259999999999994e-06 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025053+0000", 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0140000000000001e-06 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033067+0000", 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080137490000000006 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033146+0000", 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9068999999999998e-05 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033263+0000", 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117516 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034798+0000", 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001535177 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036385+0000", 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015866509999999999 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036425+0000", 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0417999999999998e-05 2024-09-16T13:16:59.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036457+0000", 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1185e-05 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:16:59.898 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.852093574000001, 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013323559, 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 46 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:16:59.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:16:59.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025065+0000", 2024-09-16T13:16:59.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7299999999999996e-06 2024-09-16T13:16:59.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025076+0000", 2024-09-16T13:16:59.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0943e-05 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025081+0000", 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.48e-06 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033698+0000", 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086164589999999999 2024-09-16T13:16:59.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033803+0000", 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010459300000000001 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033994+0000", 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000191022 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037370+0000", 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033765589999999999 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038307+0000", 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093719199999999999 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038346+0000", 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9033999999999999e-05 2024-09-16T13:16:59.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038383+0000", 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6831e-05 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.852063658999999, 2024-09-16T13:16:59.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014352281999999999, 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 47 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0699999999999996e-07 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025220+0000", 2024-09-16T13:16:59.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012995699999999999 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025231+0000", 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1297e-05 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025238+0000", 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.263e-06 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034888+0000", 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0096507490000000001 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035006+0000", 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117416 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035143+0000", 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013723699999999999 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037388+0000", 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022444710000000001 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039332+0000", 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019441930000000001 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039397+0000", 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5289000000000005e-05 2024-09-16T13:16:59.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039442+0000", 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4703000000000002e-05 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025246+0000", 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.851907191999999, 2024-09-16T13:16:59.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014350958, 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 48 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025246+0000", 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025246+0000", 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025247+0000", 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7000000000000003e-07 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025252+0000", 2024-09-16T13:16:59.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6670000000000004e-06 2024-09-16T13:16:59.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025260+0000", 2024-09-16T13:16:59.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6989999999999992e-06 2024-09-16T13:16:59.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025265+0000", 2024-09-16T13:16:59.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0300000000000001e-06 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035652+0000", 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010386687 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035734+0000", 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2108999999999996e-05 2024-09-16T13:16:59.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035898+0000", 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016392499999999999 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039267+0000", 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033692409999999998 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039511+0000", 2024-09-16T13:16:59.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024390499999999999 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039561+0000", 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9339000000000003e-05 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039597+0000", 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6386e-05 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025294+0000", 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.851858870000001, 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015060367999999999, 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 49 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025294+0000", 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025294+0000", 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025295+0000", 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025301+0000", 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5899999999999998e-06 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025356+0000", 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5495000000000003e-05 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025366+0000", 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0047000000000001e-05 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036507+0000", 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011140768000000001 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036635+0000", 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128458 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036907+0000", 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0002719 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039665+0000", 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027574269999999998 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040255+0000", 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00058990500000000005 2024-09-16T13:16:59.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040314+0000", 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9301e-05 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040355+0000", 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0717000000000003e-05 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.851767618, 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.925 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:16:59.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.851620212, 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:16:59.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:16:59.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:16:59.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:16:59.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:16:59.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:16:59.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:16:59.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:16:59.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:16:59.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:16:59.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:16:59.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:16:59.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:16:59.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:16:59.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:16:59.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:16:59.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:16:59.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:16:59.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:16:59.938 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.851592869000001, 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583774+0000", 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.293379210000001, 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011465823999999999, 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.944 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 69 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583774+0000", 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583774+0000", 2024-09-16T13:16:59.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583775+0000", 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583847+0000", 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2051999999999994e-05 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583903+0000", 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6165000000000002e-05 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583923+0000", 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9979999999999998e-05 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.589618+0000", 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056950400000000002 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.589858+0000", 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000239764 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.590239+0000", 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038074000000000002 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595048+0000", 2024-09-16T13:16:59.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004809603 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595130+0000", 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1903999999999994e-05 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595195+0000", 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4491000000000003e-05 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595240+0000", 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5034999999999998e-05 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583939+0000", 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.293214330000001, 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013326655999999999, 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.950 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 70 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583939+0000", 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583939+0000", 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583940+0000", 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6300000000000004e-07 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583945+0000", 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583956+0000", 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1147e-05 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583962+0000", 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5559999999999998e-06 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.592841+0000", 2024-09-16T13:16:59.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0088788419999999996 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.592926+0000", 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5424000000000001e-05 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.593071+0000", 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145093 2024-09-16T13:16:59.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596234+0000", 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031626990000000002 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597165+0000", 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093124799999999997 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597219+0000", 2024-09-16T13:16:59.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3680000000000001e-05 2024-09-16T13:16:59.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597266+0000", 2024-09-16T13:16:59.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6801000000000001e-05 2024-09-16T13:16:59.956 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.956 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583971+0000", 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.293182201, 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01344849, 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 71 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583971+0000", 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583971+0000", 2024-09-16T13:16:59.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583972+0000", 2024-09-16T13:16:59.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1399999999999996e-07 2024-09-16T13:16:59.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583978+0000", 2024-09-16T13:16:59.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.37e-06 2024-09-16T13:16:59.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583987+0000", 2024-09-16T13:16:59.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7360000000000007e-06 2024-09-16T13:16:59.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583992+0000", 2024-09-16T13:16:59.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1229999999999999e-06 2024-09-16T13:16:59.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.593635+0000", 2024-09-16T13:16:59.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0096431659999999999 2024-09-16T13:16:59.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.593847+0000", 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021216400000000001 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.594195+0000", 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034803999999999998 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597283+0000", 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030879169999999999 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597330+0000", 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6465999999999999e-05 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597380+0000", 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0698999999999997e-05 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597419+0000", 2024-09-16T13:16:59.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9094999999999998e-05 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.293154508000001, 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015940290999999999, 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 72 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:16:59.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4600000000000004e-07 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584119+0000", 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120004 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584145+0000", 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5539999999999999e-05 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584155+0000", 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9729999999999992e-06 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595284+0000", 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011129139999999999 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595404+0000", 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120325 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595569+0000", 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000164556 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597886+0000", 2024-09-16T13:16:59.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023171490000000001 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599784+0000", 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001898278 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599899+0000", 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114973 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599939+0000", 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9607000000000002e-05 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584171+0000", 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.292982289000001, 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016071118999999998, 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 73 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584171+0000", 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584171+0000", 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584172+0000", 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.24e-06 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584183+0000", 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0393e-05 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584210+0000", 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7679999999999999e-05 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584221+0000", 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0583e-05 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596305+0000", 2024-09-16T13:16:59.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012084068999999999 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596419+0000", 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011364799999999999 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596560+0000", 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014121000000000001 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599720+0000", 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031602819999999999 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600159+0000", 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000438818 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600201+0000", 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2342999999999998e-05 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600242+0000", 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0852999999999997e-05 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.292917607, 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:16:59.976 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:16:59.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:16:59.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.982 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.292830015, 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:16:59.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:16:59.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:16:59.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:16:59.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:16:59.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:16:59.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:16:59.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:16:59.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:16:59.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:16:59.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:16:59.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:16:59.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:16:59.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:16:59.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:16:59.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:16:59.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.292660092999999, 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:16:59.990 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.991 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.991 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:16:59.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.991 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:16:59.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:16:59.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:16:59.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:16:59.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:16:59.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:16:59.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:16:59.996 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.996 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:92 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:36.382227+0000", 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.494925693999999, 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011280221999999999, 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:16:59.997 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382227+0000", 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382227+0000", 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382228+0000", 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9999999999999996e-07 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382243+0000", 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4286e-05 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382251+0000", 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5329999999999994e-06 2024-09-16T13:16:59.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382259+0000", 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8639999999999996e-06 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.388758+0000", 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064991240000000002 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.388838+0000", 2024-09-16T13:17:00.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9987000000000005e-05 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.388964+0000", 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012593000000000001 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.392626+0000", 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036619159999999999 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393309+0000", 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068294700000000005 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393432+0000", 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123387 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393508+0000", 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5448e-05 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:00.002 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:93 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.494882512, 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011390030000000001, 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 93 2024-09-16T13:17:00.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:00.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8400000000000003e-07 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382384+0000", 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0001128 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382396+0000", 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1987000000000001e-05 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382403+0000", 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6760000000000001e-06 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.389370+0000", 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069670080000000002 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.389449+0000", 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9448000000000002e-05 2024-09-16T13:17:00.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.389563+0000", 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011318 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.392716+0000", 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031536350000000001 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:00.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393522+0000", 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080626500000000002 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393602+0000", 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9528000000000002e-05 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393661+0000", 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8718999999999998e-05 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:00.008 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.494741675, 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013820321999999999, 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:00.009 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 94 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:00.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:00.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:17:00.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2699999999999998e-07 2024-09-16T13:17:00.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:00.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382418+0000", 2024-09-16T13:17:00.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1390000000000003e-06 2024-09-16T13:17:00.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:00.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382426+0000", 2024-09-16T13:17:00.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6770000000000001e-06 2024-09-16T13:17:00.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:00.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382435+0000", 2024-09-16T13:17:00.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1330000000000005e-06 2024-09-16T13:17:00.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.391103+0000", 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086678729999999996 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.391194+0000", 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0740000000000002e-05 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:00.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.391327+0000", 2024-09-16T13:17:00.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013291 2024-09-16T13:17:00.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:00.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.394978+0000", 2024-09-16T13:17:00.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003651502 2024-09-16T13:17:00.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.396103+0000", 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011249439999999999 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.396180+0000", 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6783e-05 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:00.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:00.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.396232+0000", 2024-09-16T13:17:00.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1693999999999997e-05 2024-09-16T13:17:00.016 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:00.016 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:00.016 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:00.016 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:00.016 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:00.016 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:00.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:00.019 INFO:teuthology.orchestra.run.smithi098.stdout:504 read 0xcae1c8f thru 0xcae8984 (0x6cf6 bytes) 2024-09-16T13:17:00.019 INFO:teuthology.orchestra.run.smithi098.stdout:505 punch from 0x15f0e1c to 0x1600956, (0xfb3a bytes) 2024-09-16T13:17:00.019 INFO:teuthology.orchestra.run.smithi098.stdout:506 trunc from 0xee00c85 to 0xc372748 2024-09-16T13:17:00.019 INFO:teuthology.orchestra.run.smithi098.stdout:507 write 0x39dc13f thru 0x39eb015 (0xeed7 bytes) 2024-09-16T13:17:00.020 INFO:teuthology.orchestra.run.smithi098.stdout:508 writesame 0xc0fad17 thru 0xc0ff254 data_size 0x1(0x453e bytes) 2024-09-16T13:17:00.020 INFO:teuthology.orchestra.run.smithi098.stdout:509 write 0x51e633e thru 0x51f22e8 (0xbfab bytes) 2024-09-16T13:17:00.020 INFO:teuthology.orchestra.run.smithi098.stdout:510 trunc from 0xc372748 to 0x4df9b68 2024-09-16T13:17:00.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:17:00.021 INFO:teuthology.orchestra.run.smithi098.stdout:512 write 0x6b93f3 thru 0x6c5f84 (0xcb92 bytes) 2024-09-16T13:17:00.022 INFO:teuthology.orchestra.run.smithi098.stdout:513 read 0x14a3e73 thru 0x14af50e (0xb69c bytes) 2024-09-16T13:17:00.022 INFO:teuthology.orchestra.run.smithi098.stdout:514 write 0x676d4b5 thru 0x676e847 (0x1393 bytes) 2024-09-16T13:17:00.023 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:00.012+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:00.023 INFO:teuthology.orchestra.run.smithi098.stdout:515 trunc from 0x676e848 to 0x925b360 2024-09-16T13:17:00.031 INFO:teuthology.orchestra.run.smithi098.stdout:516 write 0x65dfc49 thru 0x65ed151 (0xd509 bytes) 2024-09-16T13:17:00.036 INFO:teuthology.orchestra.run.smithi098.stdout:518 read 0x72b2be6 thru 0x72be9cd (0xbde8 bytes) 2024-09-16T13:17:00.036 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:00.049 INFO:teuthology.orchestra.run.smithi098.stdout:521 write 0xbb06f3d thru 0xbb168ee (0xf9b2 bytes) 2024-09-16T13:17:00.054 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:00.054 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:00.067 INFO:teuthology.orchestra.run.smithi098.stdout:525 trunc from 0xbb168ef to 0x4a47c5f 2024-09-16T13:17:00.092 INFO:teuthology.orchestra.run.smithi098.stdout:526 read 0x3351fb1 thru 0x3359c83 (0x7cd3 bytes) 2024-09-16T13:17:00.093 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:00.093 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:00.110 INFO:teuthology.orchestra.run.smithi098.stdout:530 write 0x93ac0b1 thru 0x93b023c (0x418c bytes) 2024-09-16T13:17:00.118 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:00.114+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:00.125 INFO:teuthology.orchestra.run.smithi098.stdout:532 write 0xa0d1b7c thru 0xa0d599d (0x3e22 bytes) 2024-09-16T13:17:00.142 INFO:teuthology.orchestra.run.smithi098.stdout:534 trunc from 0xa0d599e to 0x2ac7a9a 2024-09-16T13:17:00.144 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:00.144 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:00.144 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:00.144 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:00.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:00.171 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:00.179 INFO:teuthology.orchestra.run.smithi098.stdout:535 writesame 0xb32b192 thru 0xb33099c data_size 0xb(0x580b bytes) 2024-09-16T13:17:00.179 INFO:teuthology.orchestra.run.smithi098.stdout:537 write 0xc180cd thru 0xc2283e (0xa772 bytes) 2024-09-16T13:17:00.187 INFO:teuthology.orchestra.run.smithi098.stdout:539 punch from 0x8f43c2c to 0x8f50336, (0xc70a bytes) 2024-09-16T13:17:00.187 INFO:teuthology.orchestra.run.smithi098.stdout:540 write 0x66b328 thru 0x66f9bf (0x4698 bytes) 2024-09-16T13:17:00.191 INFO:teuthology.orchestra.run.smithi098.stdout:541 punch from 0x6615357 to 0x6617086, (0x1d2f bytes) 2024-09-16T13:17:00.191 INFO:teuthology.orchestra.run.smithi098.stdout:542 clone 1 order 22 su 32768 sc 7 2024-09-16T13:17:00.196 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:00.214 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:00.219 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:00.212+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:00.232 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:00.253 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:00.266 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:00.266 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:00.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:00.276 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:00.306 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:00.320 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:00.314+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:00.326 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:00.329 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:00.329 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:00.329 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:00.329 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:00.329 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:00.329 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:00.329 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:00.370 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:00.370 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:00.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:00.421 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:00.414+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:00.522 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:00.514+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:00.524 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:00.525 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:00.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:00.622 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:00.618+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:00.627 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:00.627 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:00.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:00.723 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:00.718+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:00.739 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:00.739 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:00.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:00.824 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:00.816+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:00.845 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:00.845 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:00.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:00.925 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:00.920+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:00.956 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:00.956 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:00.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:01.027 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:01.022+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:01.074 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:01.074 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:01.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:01.128 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:01.122+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:01.181 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:01.181 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:01.181 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.832077899546785 2024-09-16T13:17:01.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 0.832077899546785 2024-09-16T13:17:01.184 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:01.184 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:01.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:01.224 INFO:teuthology.orchestra.run.smithi098.stdout:543 read 0x87e9fdd thru 0x87efbf3 (0x5c17 bytes) 2024-09-16T13:17:01.224 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:01.225 INFO:teuthology.orchestra.run.smithi098.stdout:546 write 0x6faf612 thru 0x6fb2ca6 (0x3695 bytes) 2024-09-16T13:17:01.242 INFO:teuthology.orchestra.run.smithi098.stdout:547 writesame 0x3cd64e1 thru 0x3ce180a data_size 0x47(0xb32a bytes) 2024-09-16T13:17:01.242 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:01.236+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:01.251 INFO:teuthology.orchestra.run.smithi098.stdout:548 read 0x15179a5 thru 0x1519d59 (0x23b5 bytes) 2024-09-16T13:17:01.261 INFO:teuthology.orchestra.run.smithi098.stdout:549 write 0xc0a6700 thru 0xc0af0da (0x89db bytes) 2024-09-16T13:17:01.268 INFO:teuthology.orchestra.run.smithi098.stdout:551 punch from 0xb45a49a to 0xb4662b7, (0xbe1d bytes) 2024-09-16T13:17:01.268 INFO:teuthology.orchestra.run.smithi098.stdout:552 punch from 0x4887435 to 0x48943b0, (0xcf7b bytes) 2024-09-16T13:17:01.268 INFO:teuthology.orchestra.run.smithi098.stdout:553 write 0x54b547 thru 0x54d6ac (0x2166 bytes) 2024-09-16T13:17:01.296 INFO:teuthology.orchestra.run.smithi098.stdout:555 read 0x95d82c9 thru 0x95db06f (0x2da7 bytes) 2024-09-16T13:17:01.296 INFO:teuthology.orchestra.run.smithi098.stdout:556 punch from 0x5ba77f1 to 0x5bb63f4, (0xec03 bytes) 2024-09-16T13:17:01.296 INFO:teuthology.orchestra.run.smithi098.stdout:558 read 0x82e45d7 thru 0x82e8ed2 (0x48fc bytes) 2024-09-16T13:17:01.296 INFO:teuthology.orchestra.run.smithi098.stdout:559 clone 2 order 18 su 65536 sc 15 2024-09-16T13:17:01.322 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:01.322 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:01.330 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:01.324+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:01.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:01.431 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:01.424+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:01.471 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:01.471 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:01.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:01.532 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:01.526+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:01.589 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:01.589 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:01.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:01.633 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:01.628+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:01.699 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:01.699 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:01.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:01.734 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:01.730+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:01.806 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:01.806 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:01.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:01.835 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:01.828+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:01.936 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:01.928+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:01.960 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:01.960 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:01.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.0 to 0.832078 (d503) 2024-09-16T13:17:02.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:17:02.037 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:02.030+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:02.072 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:02.073 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:02.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:02.138 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:02.134+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:02.232 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:02.232 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:02.239 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:02.232+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:02.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:02.340 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:02.340 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:02.341 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:02.336+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:02.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:02.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:02.442 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:02.436+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:02.446 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:02.447 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:02.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:02.481 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:02.505 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:02.527 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:02.543 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:02.538+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:02.549 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:02.554 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:02.555 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:02.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:02.572 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:02.591 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:02.610 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:02.631 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:02.635 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:02.635 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:02.635 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:02.635 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:02.635 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:02.635 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:02.643 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:02.638+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:02.666 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:02.667 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:02.667 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:02.667 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:02.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:02.744 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:02.738+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:02.770 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:02.771 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:02.771 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:02.771 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:02.771 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:02.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:02.845 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:02.838+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:02.929 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.024918+0000", 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.896929721999999, 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0098686339999999994, 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.930 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 44 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024918+0000", 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024918+0000", 2024-09-16T13:17:02.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024919+0000", 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.093e-06 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.024997+0000", 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8595000000000002e-05 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025009+0000", 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2006e-05 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025016+0000", 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999999e-06 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.031767+0000", 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067515229999999997 2024-09-16T13:17:02.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.031892+0000", 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012455100000000001 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.032088+0000", 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000196152 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034632+0000", 2024-09-16T13:17:02.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025435430000000001 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034658+0000", 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6784999999999999e-05 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034734+0000", 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5482000000000007e-05 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034786+0000", 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2173999999999999e-05 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025030+0000", 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.896817134999999, 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011426475, 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.936 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025030+0000", 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025030+0000", 2024-09-16T13:17:02.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025031+0000", 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0100000000000001e-06 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025037+0000", 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4600000000000002e-06 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025046+0000", 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2259999999999994e-06 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025053+0000", 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0140000000000001e-06 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033067+0000", 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080137490000000006 2024-09-16T13:17:02.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033146+0000", 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9068999999999998e-05 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033263+0000", 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117516 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034798+0000", 2024-09-16T13:17:02.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001535177 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036385+0000", 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015866509999999999 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036425+0000", 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0417999999999998e-05 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036457+0000", 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1185e-05 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.896787615000001, 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013323559, 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 46 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025060+0000", 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025065+0000", 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7299999999999996e-06 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025076+0000", 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0943e-05 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025081+0000", 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.48e-06 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033698+0000", 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086164589999999999 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033803+0000", 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010459300000000001 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.033994+0000", 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000191022 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037370+0000", 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033765589999999999 2024-09-16T13:17:02.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038307+0000", 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093719199999999999 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038346+0000", 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9033999999999999e-05 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038383+0000", 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6831e-05 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.896757699999998, 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014352281999999999, 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 47 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:17:02.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025090+0000", 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0699999999999996e-07 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025220+0000", 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012995699999999999 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025231+0000", 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1297e-05 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025238+0000", 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.263e-06 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.034888+0000", 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0096507490000000001 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035006+0000", 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117416 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035143+0000", 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013723699999999999 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037388+0000", 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022444710000000001 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039332+0000", 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019441930000000001 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039397+0000", 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5289000000000005e-05 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039442+0000", 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4703000000000002e-05 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.953 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025246+0000", 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.896601232999998, 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014350958, 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 48 2024-09-16T13:17:02.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025246+0000", 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025246+0000", 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025247+0000", 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7000000000000003e-07 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025252+0000", 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6670000000000004e-06 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025260+0000", 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6989999999999992e-06 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025265+0000", 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0300000000000001e-06 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035652+0000", 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010386687 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035734+0000", 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2108999999999996e-05 2024-09-16T13:17:02.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.035898+0000", 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016392499999999999 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039267+0000", 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033692409999999998 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039511+0000", 2024-09-16T13:17:02.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024390499999999999 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039561+0000", 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9339000000000003e-05 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039597+0000", 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6386e-05 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025294+0000", 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.896552911000001, 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015060367999999999, 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 49 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.960 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025294+0000", 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025294+0000", 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025295+0000", 2024-09-16T13:17:02.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025301+0000", 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5899999999999998e-06 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025356+0000", 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5495000000000003e-05 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025366+0000", 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0047000000000001e-05 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036507+0000", 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011140768000000001 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036635+0000", 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128458 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.036907+0000", 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0002719 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039665+0000", 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027574269999999998 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040255+0000", 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00058990500000000005 2024-09-16T13:17:02.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040314+0000", 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9301e-05 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040355+0000", 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0717000000000003e-05 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.896461659, 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.966 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:17:02.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:17:02.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.896314253, 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:17:02.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:17:02.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.977 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.896286910000001, 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:17:02.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:17:02.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:17:02.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:17:02.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:17:02.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:17:02.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:17:02.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583774+0000", 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.338073251000001, 2024-09-16T13:17:02.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011465823999999999, 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 69 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583774+0000", 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583774+0000", 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583775+0000", 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583847+0000", 2024-09-16T13:17:02.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2051999999999994e-05 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583903+0000", 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6165000000000002e-05 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583923+0000", 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9979999999999998e-05 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.589618+0000", 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056950400000000002 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.589858+0000", 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000239764 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.590239+0000", 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038074000000000002 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595048+0000", 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004809603 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595130+0000", 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1903999999999994e-05 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595195+0000", 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4491000000000003e-05 2024-09-16T13:17:02.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595240+0000", 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5034999999999998e-05 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583939+0000", 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.337908371000001, 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013326655999999999, 2024-09-16T13:17:02.990 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 70 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583939+0000", 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583939+0000", 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583940+0000", 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6300000000000004e-07 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583945+0000", 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583956+0000", 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1147e-05 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583962+0000", 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5559999999999998e-06 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.592841+0000", 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0088788419999999996 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.592926+0000", 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5424000000000001e-05 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.593071+0000", 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145093 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596234+0000", 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031626990000000002 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597165+0000", 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093124799999999997 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597219+0000", 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3680000000000001e-05 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597266+0000", 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6801000000000001e-05 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583971+0000", 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.337876242, 2024-09-16T13:17:02.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01344849, 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 71 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583971+0000", 2024-09-16T13:17:02.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:02.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583971+0000", 2024-09-16T13:17:02.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:02.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:02.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583972+0000", 2024-09-16T13:17:02.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1399999999999996e-07 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583978+0000", 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.37e-06 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583987+0000", 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7360000000000007e-06 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:02.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583992+0000", 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1229999999999999e-06 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.593635+0000", 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0096431659999999999 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.593847+0000", 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021216400000000001 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.594195+0000", 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034803999999999998 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597283+0000", 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030879169999999999 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597330+0000", 2024-09-16T13:17:03.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6465999999999999e-05 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597380+0000", 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0698999999999997e-05 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597419+0000", 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9094999999999998e-05 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.337848549, 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015940290999999999, 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 72 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:03.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.583999+0000", 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4600000000000004e-07 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584119+0000", 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120004 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584145+0000", 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5539999999999999e-05 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584155+0000", 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9729999999999992e-06 2024-09-16T13:17:03.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595284+0000", 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011129139999999999 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595404+0000", 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120325 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:03.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.595569+0000", 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000164556 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597886+0000", 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023171490000000001 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599784+0000", 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001898278 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:03.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599899+0000", 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114973 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599939+0000", 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9607000000000002e-05 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584171+0000", 2024-09-16T13:17:03.008 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.337676330000001, 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016071118999999998, 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 73 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584171+0000", 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584171+0000", 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584172+0000", 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.24e-06 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584183+0000", 2024-09-16T13:17:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0393e-05 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584210+0000", 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7679999999999999e-05 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584221+0000", 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0583e-05 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596305+0000", 2024-09-16T13:17:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012084068999999999 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596419+0000", 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011364799999999999 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.596560+0000", 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014121000000000001 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599720+0000", 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031602819999999999 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600159+0000", 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000438818 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600201+0000", 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2342999999999998e-05 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600242+0000", 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0852999999999997e-05 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.337611647999999, 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:17:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.337524055999999, 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:17:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:17:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.337354134000002, 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:17:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:92 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:36.382227+0000", 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 26.539619734999999, 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011280221999999999, 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:03.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382227+0000", 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382227+0000", 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382228+0000", 2024-09-16T13:17:03.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9999999999999996e-07 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382243+0000", 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4286e-05 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382251+0000", 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5329999999999994e-06 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382259+0000", 2024-09-16T13:17:03.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8639999999999996e-06 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.388758+0000", 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064991240000000002 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.388838+0000", 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9987000000000005e-05 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.388964+0000", 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012593000000000001 2024-09-16T13:17:03.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.392626+0000", 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036619159999999999 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393309+0000", 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068294700000000005 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:03.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393432+0000", 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123387 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393508+0000", 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5448e-05 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:93 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:17:03.037 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 26.539576553, 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011390030000000001, 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 93 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:17:03.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382271+0000", 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8400000000000003e-07 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382384+0000", 2024-09-16T13:17:03.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0001128 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382396+0000", 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1987000000000001e-05 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382403+0000", 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6760000000000001e-06 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:03.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.389370+0000", 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069670080000000002 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.389449+0000", 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9448000000000002e-05 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.389563+0000", 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011318 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.392716+0000", 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031536350000000001 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393522+0000", 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080626500000000002 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393602+0000", 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9528000000000002e-05 2024-09-16T13:17:03.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.393661+0000", 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8718999999999998e-05 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 26.539435716, 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013820321999999999, 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:03.043 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 94 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:03.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382412+0000", 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2699999999999998e-07 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382418+0000", 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1390000000000003e-06 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382426+0000", 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6770000000000001e-06 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.382435+0000", 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1330000000000005e-06 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.391103+0000", 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086678729999999996 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.391194+0000", 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0740000000000002e-05 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.391327+0000", 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013291 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.394978+0000", 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003651502 2024-09-16T13:17:03.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.396103+0000", 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011249439999999999 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.396180+0000", 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6783e-05 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:36.396232+0000", 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1693999999999997e-05 2024-09-16T13:17:03.048 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.049 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:03.049 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.049 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:03.049 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:03.049 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:03.051 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:17:03.051 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone1 from 0xc0af0db (overlap 0xb33099d) to 0x23d6852 2024-09-16T13:17:03.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:03.052 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:03.044+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:03.151 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:03.146+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:03.167 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:03.167 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:03.167 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:03.167 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:03.170 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #0, image image_client.0 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent1 2024-09-16T13:17:03.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:03.251 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:03.244+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:03.272 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:03.272 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:03.273 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:03.273 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:03.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:03.352 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:03.346+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:03.377 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:03.377 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:03.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:03.377 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:03.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:51 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'33, mlcod=16'27)", 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.331664680999999, 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065364949999999998, 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038562+0000", 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0920000000000001e-06 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038571+0000", 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1339999999999993e-06 2024-09-16T13:17:03.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038600+0000", 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9254000000000001e-05 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038609+0000", 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6049999999999998e-06 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038982+0000", 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037359699999999998 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039013+0000", 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0945000000000003e-05 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045052+0000", 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060389490000000001 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045097+0000", 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4919000000000001e-05 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:52 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'34, mlcod=16'29)", 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.039817+0000", 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.330408835, 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057720159999999996, 2024-09-16T13:17:03.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039817+0000", 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039817+0000", 2024-09-16T13:17:03.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039818+0000", 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.223e-06 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039828+0000", 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0125e-05 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039851+0000", 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3485999999999998e-05 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039861+0000", 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909999999999996e-06 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039933+0000", 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1829999999999995e-05 2024-09-16T13:17:03.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039967+0000", 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4375999999999997e-05 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045505+0000", 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055381470000000002 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045589+0000", 2024-09-16T13:17:03.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3537999999999999e-05 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:54 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'36, mlcod=16'33)", 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.047292+0000", 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.322933024000001, 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042292679999999996, 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047292+0000", 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047292+0000", 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047293+0000", 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:17:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047444+0000", 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000151345 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047455+0000", 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1011e-05 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047463+0000", 2024-09-16T13:17:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3159999999999999e-06 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047531+0000", 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8484000000000004e-05 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.047604+0000", 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3034999999999994e-05 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.051446+0000", 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038415739999999999 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.051522+0000", 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5891999999999998e-05 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:55 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:17:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.322092897000001, 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062294560000000004, 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:17:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5000000000000002e-07 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048240+0000", 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107209 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048252+0000", 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1263999999999999e-05 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048263+0000", 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1685000000000001e-05 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048355+0000", 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2033999999999993e-05 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048378+0000", 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3062e-05 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054305+0000", 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059262639999999997 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054362+0000", 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7287999999999998e-05 2024-09-16T13:17:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:56 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'38, mlcod=16'33)", 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.321376878999999, 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055458679999999998, 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0800000000000004e-07 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048942+0000", 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2954000000000005e-05 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048954+0000", 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2234e-05 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048961+0000", 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0530000000000003e-06 2024-09-16T13:17:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.049066+0000", 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105079 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.049088+0000", 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2427999999999999e-05 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054366+0000", 2024-09-16T13:17:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052782189999999998 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054394+0000", 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8192999999999999e-05 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:65 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'43, mlcod=16'40)", 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.584665+0000", 2024-09-16T13:17:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.785560342, 2024-09-16T13:17:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003952952, 2024-09-16T13:17:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584665+0000", 2024-09-16T13:17:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584665+0000", 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584667+0000", 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.717e-06 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584679+0000", 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2533e-05 2024-09-16T13:17:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584710+0000", 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0648999999999997e-05 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584721+0000", 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.114e-05 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584832+0000", 2024-09-16T13:17:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110697 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.584914+0000", 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2504999999999997e-05 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.588521+0000", 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036070759999999999 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.588618+0000", 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6634999999999999e-05 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:66 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'44, mlcod=16'40)", 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.585658+0000", 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.784567513999999, 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004136504, 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585658+0000", 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585658+0000", 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585660+0000", 2024-09-16T13:17:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.903e-06 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585780+0000", 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000120041 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585847+0000", 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7504999999999998e-05 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585863+0000", 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5546e-05 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.585961+0000", 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7794999999999995e-05 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.586018+0000", 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6888999999999997e-05 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.589719+0000", 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037009669999999999 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.589794+0000", 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5858000000000005e-05 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:77 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'55, mlcod=16'53)", 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.602312+0000", 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.767913074999999, 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034289580000000002, 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602312+0000", 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602312+0000", 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602313+0000", 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0330000000000001e-06 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602379+0000", 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5840000000000007e-05 2024-09-16T13:17:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602399+0000", 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9738999999999999e-05 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602406+0000", 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8929999999999999e-06 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602489+0000", 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3065000000000005e-05 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.602535+0000", 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5728e-05 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605654+0000", 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031193480000000001 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605741+0000", 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7311999999999993e-05 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:79 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'57, mlcod=16'53)", 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.603581+0000", 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.766644309, 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041617829999999996, 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603581+0000", 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603581+0000", 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603582+0000", 2024-09-16T13:17:03.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603658+0000", 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5376999999999993e-05 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603729+0000", 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1766000000000004e-05 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.603737+0000", 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6340000000000001e-06 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604193+0000", 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045645800000000003 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604248+0000", 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4242e-05 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.607658+0000", 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003410214 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.607743+0000", 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5092000000000005e-05 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:80 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-16T13:17:03.418 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.604358+0000", 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.765867239999999, 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060398550000000002, 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604358+0000", 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604358+0000", 2024-09-16T13:17:03.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604359+0000", 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.054e-06 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604451+0000", 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1916000000000006e-05 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604469+0000", 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7376000000000001e-05 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604477+0000", 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0870000000000007e-06 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605124+0000", 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00064696399999999998 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605155+0000", 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1259e-05 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.610343+0000", 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051882860000000003 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.610398+0000", 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4913000000000001e-05 2024-09-16T13:17:03.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:90 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'64, mlcod=16'60)", 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:36.385144+0000", 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 26.985081542, 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033364800000000002, 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385144+0000", 2024-09-16T13:17:03.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385144+0000", 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385146+0000", 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.643e-06 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385337+0000", 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019110199999999999 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385364+0000", 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7265999999999999e-05 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385375+0000", 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.147e-05 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385455+0000", 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9265999999999996e-05 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.385481+0000", 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6752999999999999e-05 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388397+0000", 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029152079999999999 2024-09-16T13:17:03.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388480+0000", 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3771999999999997e-05 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:91 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'65, mlcod=16'60)", 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:36.387259+0000", 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 26.982966566999998, 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038175589999999999, 2024-09-16T13:17:03.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387259+0000", 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387259+0000", 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387260+0000", 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.082e-06 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387268+0000", 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7239999999999992e-06 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387284+0000", 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6566999999999999e-05 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387293+0000", 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3599999999999996e-06 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387380+0000", 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7008000000000004e-05 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.387419+0000", 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9628000000000001e-05 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.390959+0000", 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003539964 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391076+0000", 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011722600000000001 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:92 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'66, mlcod=17'62)", 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:36.388758+0000", 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 26.981467381000002, 2024-09-16T13:17:03.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032607000000000001, 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388758+0000", 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388758+0000", 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388759+0000", 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1319999999999999e-06 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388769+0000", 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6560000000000004e-06 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388792+0000", 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3536e-05 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388802+0000", 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1430000000000004e-06 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.388967+0000", 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016558999999999999 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.389011+0000", 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3451999999999997e-05 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.391957+0000", 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00294655 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.392019+0000", 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1641000000000002e-05 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:96 1.0 e17/15 1:30306672:devicehealth::main.db.0000000000000000:head v 17'70, mlcod=17'67)", 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:36.397712+0000", 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 26.972513882000001, 2024-09-16T13:17:03.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033433479999999999, 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397712+0000", 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397712+0000", 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397713+0000", 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-06 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397789+0000", 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5776000000000001e-05 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397816+0000", 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7114000000000002e-05 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397833+0000", 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6748999999999998e-05 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397903+0000", 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9935e-05 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.397930+0000", 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.794e-05 2024-09-16T13:17:03.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.401013+0000", 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030821830000000001 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:36.401055+0000", 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2299000000000003e-05 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1221 3.17 3:e95d8979:::rbd_data.112de98401a1.000000000000001a:head [set-alloc-hint object_size 4194304 write_size 4194304,write 4122414~58735] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.299762+0000", 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.0704633509999999, 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033736249999999999, 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1221 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.299762+0000", 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.299762+0000", 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.299765+0000", 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9170000000000002e-06 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.299859+0000", 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4197000000000006e-05 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.299896+0000", 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6806e-05 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.299909+0000", 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3416e-05 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.300012+0000", 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102487 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.300151+0000", 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000138663 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.300407+0000", 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025613699999999998 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.302000+0000", 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015937449999999999 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.303001+0000", 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001000637 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.303073+0000", 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2323000000000005e-05 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.303136+0000", 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2297000000000003e-05 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1265 3.1f 3:fb65d5ef:::rbd_data.112de98401a1.000000000000000a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.500330+0000", 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.8698959029999997, 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040623589999999998, 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.445 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1265 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500330+0000", 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500330+0000", 2024-09-16T13:17:03.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500332+0000", 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6019999999999998e-06 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500344+0000", 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1983e-05 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500378+0000", 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4214000000000001e-05 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500391+0000", 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3039e-05 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500455+0000", 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3912999999999996e-05 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500543+0000", 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7216e-05 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500628+0000", 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5144999999999999e-05 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.502048+0000", 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001420389 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.504255+0000", 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022069559999999999 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.504327+0000", 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1557999999999995e-05 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.504392+0000", 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5344000000000003e-05 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.450 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1266 3.1 3:8739a53c:::rbd_data.112de98401a1.000000000000000b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.500416+0000", 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.8698097479999998, 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055098930000000001, 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1266 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.451 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500416+0000", 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500416+0000", 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500417+0000", 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1829999999999999e-06 2024-09-16T13:17:03.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500425+0000", 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9880000000000004e-06 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500441+0000", 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5614e-05 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500451+0000", 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0071000000000001e-05 2024-09-16T13:17:03.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500577+0000", 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012604700000000001 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500769+0000", 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019282699999999999 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501049+0000", 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027978600000000002 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503155+0000", 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021059630000000002 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.505807+0000", 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026514199999999998 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.505880+0000", 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2960999999999995e-05 2024-09-16T13:17:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.505926+0000", 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6032999999999998e-05 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1270 3.17 3:e95d8979:::rbd_data.112de98401a1.000000000000001a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.500499+0000", 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.8697264100000002, 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003880465, 2024-09-16T13:17:03.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1270 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500499+0000", 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500499+0000", 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500500+0000", 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3629999999999999e-06 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500507+0000", 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2879999999999998e-06 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500523+0000", 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5985999999999999e-05 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500533+0000", 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9790000000000005e-06 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500674+0000", 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014117099999999999 2024-09-16T13:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500863+0000", 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018922500000000001 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501134+0000", 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027073700000000001 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.502510+0000", 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013766729999999999 2024-09-16T13:17:03.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.504279+0000", 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001768459 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.504332+0000", 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3075999999999998e-05 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.504380+0000", 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7508e-05 2024-09-16T13:17:03.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1858 3.17 3:efe05268:::rbd_data.112d2ce070e7.0000000000000014:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:02.992700+0000", 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.37752538600000002, 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059941489999999998, 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1858 2024-09-16T13:17:03.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992700+0000", 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992700+0000", 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992703+0000", 2024-09-16T13:17:03.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3859999999999999e-06 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992714+0000", 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0396e-05 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992747+0000", 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2753000000000003e-05 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992776+0000", 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9159e-05 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992869+0000", 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3700000000000001e-05 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.993141+0000", 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027109399999999999 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.998694+0000", 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055536609999999997 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1869 3.17 3:efe05268:::rbd_data.112d2ce070e7.0000000000000014:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:02.999754+0000", 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.370471457, 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041195090000000004, 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1869 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.999754+0000", 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.999754+0000", 2024-09-16T13:17:03.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.999757+0000", 2024-09-16T13:17:03.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5720000000000001e-06 2024-09-16T13:17:03.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.999767+0000", 2024-09-16T13:17:03.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0216e-05 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.999795+0000", 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7951000000000001e-05 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.999806+0000", 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.132e-05 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.999872+0000", 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5984999999999999e-05 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.999943+0000", 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0570999999999999e-05 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.000202+0000", 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025921499999999998 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.001960+0000", 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017581770000000001 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.003780+0000", 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018195990000000001 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.003835+0000", 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5383999999999997e-05 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.003874+0000", 2024-09-16T13:17:03.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8519000000000002e-05 2024-09-16T13:17:03.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.472 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:03.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:03.474 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:03.468+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:03.541 INFO:teuthology.orchestra.run.smithi098.stdout:560 read 0x1f58837 thru 0x1f5f740 (0x6f0a bytes) 2024-09-16T13:17:03.542 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:03.542 INFO:teuthology.orchestra.run.smithi098.stdout:564 read 0x9780df0 thru 0x978946b (0x867c bytes) 2024-09-16T13:17:03.543 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:03.543 INFO:teuthology.orchestra.run.smithi098.stdout:566 punch from 0x39e882f to 0x39eda8e, (0x525f bytes) 2024-09-16T13:17:03.554 INFO:teuthology.orchestra.run.smithi098.stdout:567 trunc from 0xc0af0db to 0x925b761 2024-09-16T13:17:03.574 INFO:teuthology.orchestra.run.smithi098.stdout:569 write 0x4ff8630 thru 0x4ffff30 (0x7901 bytes) 2024-09-16T13:17:03.574 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:03.568+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:03.597 INFO:teuthology.orchestra.run.smithi098.stdout:570 writesame 0xcbdcaeb thru 0xcbe724e data_size 0x6(0xa764 bytes) 2024-09-16T13:17:03.598 INFO:teuthology.orchestra.run.smithi098.stdout:571 read 0x98d2433 thru 0x98dbc85 (0x9853 bytes) 2024-09-16T13:17:03.598 INFO:teuthology.orchestra.run.smithi098.stdout:572 write 0xc23037e thru 0xc23a29d (0x9f20 bytes) 2024-09-16T13:17:03.603 INFO:teuthology.orchestra.run.smithi098.stdout:573 writesame 0x63dd6b4 thru 0x63e9d83 data_size 0x8(0xc6d0 bytes) 2024-09-16T13:17:03.616 INFO:teuthology.orchestra.run.smithi098.stdout:574 writesame 0x22252ae thru 0x22271f9 data_size 0x2(0x1f4c bytes) 2024-09-16T13:17:03.625 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:03.625 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:03.625 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:03.625 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:03.625 INFO:teuthology.orchestra.run.smithi098.stdout:575 writesame 0x8efe099 thru 0x8f0a3b6 data_size 0x4b(0xc31e bytes) 2024-09-16T13:17:03.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:03.635 INFO:teuthology.orchestra.run.smithi098.stdout:576 write 0x66949a2 thru 0x669b8fe (0x6f5d bytes) 2024-09-16T13:17:03.643 INFO:teuthology.orchestra.run.smithi098.stdout:577 read 0x69ba1de thru 0x69bd85e (0x3681 bytes) 2024-09-16T13:17:03.643 INFO:teuthology.orchestra.run.smithi098.stdout:578 read 0x8906fe6 thru 0x8911d7d (0xad98 bytes) 2024-09-16T13:17:03.643 INFO:teuthology.orchestra.run.smithi098.stdout:579 read 0x61c4822 thru 0x61d3f80 (0xf75f bytes) 2024-09-16T13:17:03.643 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:03.652 INFO:teuthology.orchestra.run.smithi098.stdout:582 trunc from 0xcbe724f to 0x7f5d803 2024-09-16T13:17:03.675 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:03.670+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:03.683 INFO:teuthology.orchestra.run.smithi098.stdout:583 punch from 0x5da859e to 0x5db1186, (0x8be8 bytes) 2024-09-16T13:17:03.702 INFO:teuthology.orchestra.run.smithi098.stdout:584 writesame 0xb947270 thru 0xb9527f7 data_size 0x94(0xb588 bytes) 2024-09-16T13:17:03.719 INFO:teuthology.orchestra.run.smithi098.stdout:587 trunc from 0xb9527f8 to 0xcb0cab 2024-09-16T13:17:03.734 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:03.734 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:03.734 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:03.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:03.785 INFO:teuthology.orchestra.run.smithi098.stdout:588 writesame 0x15597a5 thru 0x1566391 data_size 0x35(0xcbed bytes) 2024-09-16T13:17:03.785 INFO:teuthology.orchestra.run.smithi098.stdout:590 punch from 0x456b11 to 0x459a7e, (0x2f6d bytes) 2024-09-16T13:17:03.785 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:03.785 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:03.785 INFO:teuthology.orchestra.run.smithi098.stdout:595 read 0xbec9e2 thru 0xbfb362 (0xe981 bytes) 2024-09-16T13:17:03.786 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:03.780+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:03.799 INFO:teuthology.orchestra.run.smithi098.stdout:597 write 0x2dab285 thru 0x2db198b (0x6707 bytes) 2024-09-16T13:17:03.809 INFO:teuthology.orchestra.run.smithi098.stdout:598 read 0x1ddc290 thru 0x1dddabb (0x182c bytes) 2024-09-16T13:17:03.809 INFO:teuthology.orchestra.run.smithi098.stdout:600 read 0x639669 thru 0x641c44 (0x85dc bytes) 2024-09-16T13:17:03.834 INFO:teuthology.orchestra.run.smithi098.stdout:603 writesame 0x9a32527 thru 0x9a39b92 data_size 0x6a(0x766c bytes) 2024-09-16T13:17:03.841 INFO:teuthology.orchestra.run.smithi098.stdout:605 trunc from 0x9a39b93 to 0xe060450 2024-09-16T13:17:03.853 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:03.853 INFO:teuthology.orchestra.run.smithi098.stdout:611 punch from 0xcb6ade4 to 0xcb6fe3e, (0x505a bytes) 2024-09-16T13:17:03.853 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:03.853 INFO:teuthology.orchestra.run.smithi098.stdout:616 punch from 0x6db9266 to 0x6db9392, (0x12c bytes) 2024-09-16T13:17:03.853 INFO:teuthology.orchestra.run.smithi098.stdout:618 writesame 0x8f7e7af thru 0x8f7f23a data_size 0x14(0xa8c bytes) 2024-09-16T13:17:03.861 INFO:teuthology.orchestra.run.smithi098.stdout:619 write 0x1c2cebc thru 0x1c3b0a4 (0xe1e9 bytes) 2024-09-16T13:17:03.871 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:03.871 INFO:teuthology.orchestra.run.smithi098.stdout:622 punch from 0x17b2799 to 0x17b9b99, (0x7400 bytes) 2024-09-16T13:17:03.871 INFO:teuthology.orchestra.run.smithi098.stdout:624 writesame 0x706e689 thru 0x707da48 data_size 0xc0(0xf3c0 bytes) 2024-09-16T13:17:03.876 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:03.872+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:03.879 INFO:teuthology.orchestra.run.smithi098.stdout:625 punch from 0x2f5084c to 0x2f52151, (0x1905 bytes) 2024-09-16T13:17:03.879 INFO:teuthology.orchestra.run.smithi098.stdout:626 read 0x4881354 thru 0x4889b94 (0x8841 bytes) 2024-09-16T13:17:03.879 INFO:teuthology.orchestra.run.smithi098.stdout:627 read 0xa04ff45 thru 0xa05f04e (0xf10a bytes) 2024-09-16T13:17:03.879 INFO:teuthology.orchestra.run.smithi098.stdout:628 write 0x80c46fb thru 0x80ca9a0 (0x62a6 bytes) 2024-09-16T13:17:03.885 INFO:teuthology.orchestra.run.smithi098.stdout:630 write 0x9cda8ee thru 0x9ce167d (0x6d90 bytes) 2024-09-16T13:17:03.893 INFO:teuthology.orchestra.run.smithi098.stdout:631 write 0xb694b thru 0xc4053 (0xd709 bytes) 2024-09-16T13:17:03.902 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4281.0:7 2.2 e21/18 2:5cea7035:::rbd_info:head v 21'2, mlcod=0'0)", 2024-09-16T13:17:03.902 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:44.841612+0000", 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.054702468999999, 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021828479999999998, 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841612+0000", 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841612+0000", 2024-09-16T13:17:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841613+0000", 2024-09-16T13:17:03.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841622+0000", 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.935e-06 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841645+0000", 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3686e-05 2024-09-16T13:17:03.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841658+0000", 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2279999999999999e-05 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841760+0000", 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102268 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.841830+0000", 2024-09-16T13:17:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0000999999999996e-05 2024-09-16T13:17:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.843740+0000", 2024-09-16T13:17:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019105789999999999 2024-09-16T13:17:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:44.843795+0000", 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4691999999999997e-05 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:9 3.1a e26/23 3:5cea7035:::rbd_info:head v 26'2, mlcod=0'0)", 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:54.303946+0000", 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.5923684310000006, 2024-09-16T13:17:03.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002066656, 2024-09-16T13:17:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.303946+0000", 2024-09-16T13:17:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.303946+0000", 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.303947+0000", 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.303954+0000", 2024-09-16T13:17:03.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6090000000000003e-06 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.304014+0000", 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9966000000000002e-05 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.304022+0000", 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5209999999999997e-06 2024-09-16T13:17:03.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.304102+0000", 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0642999999999993e-05 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.304129+0000", 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.639e-05 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.305959+0000", 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018306329999999999 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:54.306013+0000", 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3514999999999998e-05 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:630 3.0 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1922377~21157] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:57.780201+0000", 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.1161136389999999, 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024889439999999999, 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.914 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 630 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.780201+0000", 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.780201+0000", 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.780203+0000", 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2400000000000002e-06 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.780225+0000", 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1878999999999998e-05 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.780250+0000", 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4522999999999999e-05 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.780260+0000", 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.008e-05 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.780333+0000", 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3236000000000002e-05 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.780463+0000", 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013002599999999999 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.780704+0000", 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000241146 2024-09-16T13:17:03.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.781948+0000", 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001243669 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.782547+0000", 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00059891199999999995 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.782634+0000", 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6798999999999993e-05 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:57.782690+0000", 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6434000000000002e-05 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:906 3.0 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3207221~60502] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.428393+0000", 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.467922068, 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029788089999999998, 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.921 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 906 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428393+0000", 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428393+0000", 2024-09-16T13:17:03.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428395+0000", 2024-09-16T13:17:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.48e-06 2024-09-16T13:17:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428490+0000", 2024-09-16T13:17:03.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5104000000000003e-05 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428520+0000", 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9454e-05 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428532+0000", 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2512e-05 2024-09-16T13:17:03.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428595+0000", 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3207999999999995e-05 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428660+0000", 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4387999999999994e-05 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428778+0000", 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118449 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.430005+0000", 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001226352 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.431266+0000", 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001261135 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.431328+0000", 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2359000000000004e-05 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.431371+0000", 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3368000000000002e-05 2024-09-16T13:17:03.927 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:989 3.0 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head [write 704429~36241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.646990+0000", 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.2493250600000003, 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028331820000000001, 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.928 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 989 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.646990+0000", 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.646990+0000", 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.646992+0000", 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647014+0000", 2024-09-16T13:17:03.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2251000000000001e-05 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647035+0000", 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.092e-05 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647042+0000", 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3849999999999996e-06 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647090+0000", 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7512999999999997e-05 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647169+0000", 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9756e-05 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647279+0000", 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109954 2024-09-16T13:17:03.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.648654+0000", 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001374826 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.649755+0000", 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011004560000000001 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.649794+0000", 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9334e-05 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.649823+0000", 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8844e-05 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:990 3.0 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head [write 3828425~26124] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.650972+0000", 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.2453422390000002, 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026479530000000002, 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 990 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.650972+0000", 2024-09-16T13:17:03.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.650972+0000", 2024-09-16T13:17:03.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.650974+0000", 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.528e-06 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.650995+0000", 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0795999999999999e-05 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651018+0000", 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3351000000000001e-05 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651027+0000", 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4919999999999993e-06 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651081+0000", 2024-09-16T13:17:03.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3875999999999997e-05 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651141+0000", 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0504000000000002e-05 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651237+0000", 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6446000000000005e-05 2024-09-16T13:17:03.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.652328+0000", 2024-09-16T13:17:03.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010902959999999999 2024-09-16T13:17:03.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.653439+0000", 2024-09-16T13:17:03.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001110918 2024-09-16T13:17:03.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.653534+0000", 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5048000000000002e-05 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.653620+0000", 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6698000000000002e-05 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.941 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:992 3.0 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.659546+0000", 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.2367691580000004, 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002513253, 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 992 2024-09-16T13:17:03.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.943 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.659546+0000", 2024-09-16T13:17:03.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.659546+0000", 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.659548+0000", 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0200000000000001e-06 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.659556+0000", 2024-09-16T13:17:03.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1319999999999994e-06 2024-09-16T13:17:03.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.659574+0000", 2024-09-16T13:17:03.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8681999999999998e-05 2024-09-16T13:17:03.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.659587+0000", 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2563e-05 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.659675+0000", 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8473999999999994e-05 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.659832+0000", 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015678299999999999 2024-09-16T13:17:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.660027+0000", 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019469600000000001 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.661329+0000", 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001301612 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.661972+0000", 2024-09-16T13:17:03.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00064377 2024-09-16T13:17:03.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.662021+0000", 2024-09-16T13:17:03.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8680999999999997e-05 2024-09-16T13:17:03.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.662059+0000", 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7839999999999997e-05 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1031 3.4 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,write 972249~26653] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.776038+0000", 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.1202767270000002, 2024-09-16T13:17:03.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032075620000000002, 2024-09-16T13:17:03.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.950 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.950 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.950 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.950 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1031 2024-09-16T13:17:03.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.950 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.776038+0000", 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.776038+0000", 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.776040+0000", 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1710000000000001e-06 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.776062+0000", 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1742e-05 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.776084+0000", 2024-09-16T13:17:03.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2272999999999998e-05 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.776093+0000", 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4719999999999995e-06 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.776232+0000", 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013902700000000001 2024-09-16T13:17:03.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.776457+0000", 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000225085 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.776920+0000", 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00046334600000000001 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.778807+0000", 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001886612 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.779066+0000", 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025965800000000001 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.779160+0000", 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3974000000000005e-05 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.779246+0000", 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5202000000000002e-05 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1263 3.0 3:0214ce37:::rbd_data.112de98401a1.0000000000000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.500325+0000", 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3959899489999996, 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031122630000000001, 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1263 2024-09-16T13:17:03.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500325+0000", 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500325+0000", 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500327+0000", 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5660000000000001e-06 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500339+0000", 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1291000000000001e-05 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500374+0000", 2024-09-16T13:17:03.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5271000000000002e-05 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500390+0000", 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5682e-05 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500552+0000", 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016214699999999999 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500768+0000", 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021672600000000001 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501065+0000", 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029705900000000002 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.502291+0000", 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001225596 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503370+0000", 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010786610000000001 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503403+0000", 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2759000000000003e-05 2024-09-16T13:17:03.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503437+0000", 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4505000000000003e-05 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1264 3.4 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.500422+0000", 2024-09-16T13:17:03.963 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.395892645, 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036847239999999999, 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1264 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500422+0000", 2024-09-16T13:17:03.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500422+0000", 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500423+0000", 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9100000000000002e-07 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500427+0000", 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.34e-06 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500460+0000", 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3039e-05 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500486+0000", 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5785000000000001e-05 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500601+0000", 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011500100000000001 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500762+0000", 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016081400000000001 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500903+0000", 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014128900000000001 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501529+0000", 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00062578199999999997 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.504044+0000", 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025146180000000001 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.504082+0000", 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8732000000000001e-05 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.504107+0000", 2024-09-16T13:17:03.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4433000000000001e-05 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1466 3.8 e28/23 3:12e8453a:::rbd_data.112de98401a1.0000000000000019:head v 28'307, mlcod=28'305)", 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:00.028437+0000", 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.8678775229999998, 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002157253, 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.970 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:00.028437+0000", 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:00.028437+0000", 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:00.028438+0000", 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.189e-06 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:00.028484+0000", 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5646999999999999e-05 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:00.028507+0000", 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2963999999999999e-05 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:00.028519+0000", 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2468e-05 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:00.028625+0000", 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105405 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:00.028680+0000", 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5488999999999997e-05 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:00.030523+0000", 2024-09-16T13:17:03.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018430149999999999 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:00.030594+0000", 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1075999999999995e-05 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.975 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1821 3.0 3:04479396:::rbd_id.image_client.0-clone2:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:02.954269+0000", 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.94204566300000003, 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003636706, 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1821 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.954269+0000", 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.954269+0000", 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.954271+0000", 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2579999999999998e-06 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.954280+0000", 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.954304+0000", 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3989999999999999e-05 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.954313+0000", 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4849999999999998e-06 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.954398+0000", 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4572000000000003e-05 2024-09-16T13:17:03.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.954409+0000", 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0995000000000001e-05 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.957730+0000", 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033211880000000001 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.957822+0000", 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1632999999999995e-05 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.957906+0000", 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4226000000000003e-05 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1928 3.19 3:9b8fa907:::rbd_data.112d2ce070e7.000000000000000c:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,truncate 1173586] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.044568+0000", 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.85174712299999999, 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023229349999999999, 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1928 2024-09-16T13:17:03.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.044568+0000", 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.044568+0000", 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.044569+0000", 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6080000000000001e-06 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.044578+0000", 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6109999999999994e-06 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.044600+0000", 2024-09-16T13:17:03.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2050000000000001e-05 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.044609+0000", 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7159999999999992e-06 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.044685+0000", 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6316000000000006e-05 2024-09-16T13:17:03.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.044874+0000", 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018864000000000001 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.045119+0000", 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024541599999999998 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.046075+0000", 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00095596600000000004 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.046688+0000", 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00061324299999999997 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:03.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.046813+0000", 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000124521 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.046891+0000", 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7848000000000004e-05 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.988 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2053 3.0 3:07989a36:::rbd_data.112dbfd5d823.000000000000024e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.551297+0000", 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.34501802799999998, 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026296309999999999, 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2053 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551297+0000", 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551297+0000", 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551299+0000", 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4360000000000001e-06 2024-09-16T13:17:03.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551308+0000", 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0359999999999995e-06 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551335+0000", 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6363000000000002e-05 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551347+0000", 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2029e-05 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551499+0000", 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000152335 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551759+0000", 2024-09-16T13:17:03.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000260287 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.553926+0000", 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021671450000000001 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.993 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2058 3.4 3:27247e1b:::rbd_data.112dbfd5d823.0000000000000253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.551371+0000", 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.34494365799999999, 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022576900000000001, 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2058 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551371+0000", 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551371+0000", 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551372+0000", 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.06e-07 2024-09-16T13:17:03.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551376+0000", 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.788e-06 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551386+0000", 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.039e-05 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551389+0000", 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3100000000000001e-06 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551438+0000", 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8874e-05 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.551637+0000", 2024-09-16T13:17:03.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019877100000000001 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.553629+0000", 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019918510000000002 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2106 3.19 3:9b478818:::rbd_data.112dbfd5d823.0000000000000242:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,write 0~41911] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.628184+0000", 2024-09-16T13:17:03.998 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.26813060399999999, 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030367010000000002, 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2106 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.628184+0000", 2024-09-16T13:17:03.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.628184+0000", 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.628186+0000", 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2589999999999999e-06 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.628211+0000", 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5114999999999999e-05 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.628236+0000", 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4153000000000001e-05 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.628245+0000", 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3300000000000005e-06 2024-09-16T13:17:04.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.628324+0000", 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9252000000000006e-05 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.628539+0000", 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021468100000000001 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.628830+0000", 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029064500000000002 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.630798+0000", 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019685060000000001 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:04.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.631155+0000", 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035678899999999999 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.631199+0000", 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3764000000000003e-05 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.631221+0000", 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2206999999999999e-05 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.004 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2302 3.0 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.760814+0000", 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.13550109699999999, 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027110300000000001, 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:04.005 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2302 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.760814+0000", 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.760814+0000", 2024-09-16T13:17:04.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.760815+0000", 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.885e-06 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.760871+0000", 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6007000000000001e-05 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.760903+0000", 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1198000000000002e-05 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.760916+0000", 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3040999999999999e-05 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.761031+0000", 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011574 2024-09-16T13:17:04.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.761261+0000", 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022923900000000001 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.761477+0000", 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000216703 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.763168+0000", 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016909749999999999 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.763344+0000", 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0001761 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.763437+0000", 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2961000000000007e-05 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.763525+0000", 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7181000000000004e-05 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2298 3.8 e31/23 3:15ee1bf2:::rbd_data.112dbfd5d823.0000000000000037:head v 31'324, mlcod=29'322)", 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.761079+0000", 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.135235628, 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021486579999999999, 2024-09-16T13:17:04.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.761079+0000", 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.761079+0000", 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.761080+0000", 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0419999999999999e-06 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.761088+0000", 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7489999999999998e-06 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.761098+0000", 2024-09-16T13:17:04.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0118e-05 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.761103+0000", 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.339e-06 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.762206+0000", 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001103056 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.762242+0000", 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5775000000000002e-05 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.763171+0000", 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00092915399999999996 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.763228+0000", 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6425000000000003e-05 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.015 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2334 3.19 3:98eb4bee:::rbd_data.112dbfd5d823.00000000000000be:head [set-alloc-hint object_size 262144 write_size 262144,write 45701~19835] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.800645+0000", 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.095670190000000002, 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027723349999999999, 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:04.016 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2334 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.800645+0000", 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.800645+0000", 2024-09-16T13:17:04.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.800647+0000", 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9720000000000001e-06 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.800687+0000", 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9501e-05 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.800720+0000", 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2861999999999998e-05 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.800732+0000", 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.185e-05 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.800834+0000", 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010269500000000001 2024-09-16T13:17:04.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.801106+0000", 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027163900000000001 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.801351+0000", 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00024452000000000002 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.802867+0000", 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001516255 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.803245+0000", 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037792899999999999 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.803343+0000", 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.802e-05 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.803417+0000", 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4091999999999995e-05 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2361 3.0 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head [set-alloc-hint object_size 262144 write_size 262144,write 131072~45221] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:04.022 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.863932+0000", 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.032382755999999999, 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003083733, 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2361 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.863932+0000", 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.863932+0000", 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.863936+0000", 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5530000000000001e-06 2024-09-16T13:17:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864043+0000", 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010715199999999999 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864097+0000", 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4234999999999998e-05 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864117+0000", 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0326000000000002e-05 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864195+0000", 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8189999999999995e-05 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864364+0000", 2024-09-16T13:17:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016837199999999999 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864527+0000", 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016372000000000001 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.865735+0000", 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001207839 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:04.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.866923+0000", 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011873420000000001 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.866973+0000", 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0096000000000003e-05 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.867016+0000", 2024-09-16T13:17:04.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2908000000000003e-05 2024-09-16T13:17:04.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.029 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.029 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.029 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:04.030 INFO:teuthology.orchestra.run.smithi098.stdout:632 write 0xa59c296 thru 0xa5a68ef (0xa65a bytes) 2024-09-16T13:17:04.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:04.033 INFO:teuthology.orchestra.run.smithi098.stdout:633 read 0xbceea85 thru 0xbcf0d95 (0x2311 bytes) 2024-09-16T13:17:04.033 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.033 INFO:teuthology.orchestra.run.smithi098.stdout:635 write 0x88112fe thru 0x8812edc (0x1bdf bytes) 2024-09-16T13:17:04.033 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.033 INFO:teuthology.orchestra.run.smithi098.stdout:639 writesame 0x74034c2 thru 0x7405e15 data_size 0x2e(0x2954 bytes) 2024-09-16T13:17:04.033 INFO:teuthology.orchestra.run.smithi098.stdout:640 read 0x44680fc thru 0x4471f0e (0x9e13 bytes) 2024-09-16T13:17:04.033 INFO:teuthology.orchestra.run.smithi098.stdout:641 read 0x2323347 thru 0x2325856 (0x2510 bytes) 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:643 writesame 0x7f7ee66 thru 0x7f88a2c data_size 0x3f(0x9bc7 bytes) 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:644 read 0x8749d9f thru 0x874ae66 (0x10c8 bytes) 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:645 read 0x6de4436 thru 0x6def0fd (0xacc8 bytes) 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:646 punch from 0xbfe927 to 0xc0e0c1, (0xf79a bytes) 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:647 write 0x531da73 thru 0x5326ad0 (0x905e bytes) 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:648 write 0xd0b3cab thru 0xd0bec39 (0xaf8f bytes) 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:649 writesame 0x1432a5d thru 0x1432e96 data_size 0x1(0x43a bytes) 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:650 trunc from 0xe060450 to 0xc5d58e8 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:651 punch from 0x608b84f to 0x6098889, (0xd03a bytes) 2024-09-16T13:17:04.034 INFO:teuthology.orchestra.run.smithi098.stdout:652 read 0x1849ce0 thru 0x18594ef (0xf810 bytes) 2024-09-16T13:17:04.035 INFO:teuthology.orchestra.run.smithi098.stdout:653 read 0x3fdba7b thru 0x3fe9be4 (0xe16a bytes) 2024-09-16T13:17:04.035 INFO:teuthology.orchestra.run.smithi098.stdout:655 trunc from 0xc5d58e8 to 0x6e1148 2024-09-16T13:17:04.035 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:04.026+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:04.058 INFO:teuthology.orchestra.run.smithi098.stdout:656 writesame 0x5f0b49 thru 0x5fd810 data_size 0x4(0xccc8 bytes) 2024-09-16T13:17:04.077 INFO:teuthology.orchestra.run.smithi098.stdout:657 punch from 0x381a82 to 0x38630b, (0x4889 bytes) 2024-09-16T13:17:04.077 INFO:teuthology.orchestra.run.smithi098.stdout:659 read 0x36423b thru 0x36ae89 (0x6c4f bytes) 2024-09-16T13:17:04.078 INFO:teuthology.orchestra.run.smithi098.stdout:660 punch from 0x127cb5 to 0x133ce2, (0xc02d bytes) 2024-09-16T13:17:04.094 INFO:teuthology.orchestra.run.smithi098.stdout:661 write 0x2a7efb3 thru 0x2a8a05f (0xb0ad bytes) 2024-09-16T13:17:04.103 INFO:teuthology.orchestra.run.smithi098.stdout:662 punch from 0x2a7a6ae to 0x2a84312, (0x9c64 bytes) 2024-09-16T13:17:04.127 INFO:teuthology.orchestra.run.smithi098.stdout:663 write 0x9b5457a thru 0x9b618ca (0xd351 bytes) 2024-09-16T13:17:04.131 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:04.124+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:04.140 INFO:teuthology.orchestra.run.smithi098.stdout:665 trunc from 0x9b618cb to 0x7e05bdd 2024-09-16T13:17:04.168 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:04.168 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:04.168 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:04.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:04.180 INFO:teuthology.orchestra.run.smithi098.stdout:666 write 0xc96feb0 thru 0xc97061c (0x76d bytes) 2024-09-16T13:17:04.189 INFO:teuthology.orchestra.run.smithi098.stdout:667 punch from 0x16b4e0e to 0x16ba2a5, (0x5497 bytes) 2024-09-16T13:17:04.191 INFO:teuthology.orchestra.run.smithi098.stdout:669 trunc from 0xc97061d to 0xc12a6bc 2024-09-16T13:17:04.208 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.209 INFO:teuthology.orchestra.run.smithi098.stdout:671 read 0x4df4215 thru 0x4dfe7ec (0xa5d8 bytes) 2024-09-16T13:17:04.209 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.220 INFO:teuthology.orchestra.run.smithi098.stdout:673 writesame 0xc550a91 thru 0xc55a95c data_size 0x2(0x9ecc bytes) 2024-09-16T13:17:04.220 INFO:teuthology.orchestra.run.smithi098.stdout:676 read 0x8a3a43 thru 0x8a5d0d (0x22cb bytes) 2024-09-16T13:17:04.220 INFO:teuthology.orchestra.run.smithi098.stdout:677 punch from 0x1644325 to 0x1651306, (0xcfe1 bytes) 2024-09-16T13:17:04.220 INFO:teuthology.orchestra.run.smithi098.stdout:679 write 0x3aa8395 thru 0x3aa9b70 (0x17dc bytes) 2024-09-16T13:17:04.228 INFO:teuthology.orchestra.run.smithi098.stdout:680 write 0x9450a13 thru 0x9456573 (0x5b61 bytes) 2024-09-16T13:17:04.232 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:04.228+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:04.235 INFO:teuthology.orchestra.run.smithi098.stdout:681 punch from 0x8fe35bc to 0x8fe60e3, (0x2b27 bytes) 2024-09-16T13:17:04.235 INFO:teuthology.orchestra.run.smithi098.stdout:682 write 0x2b63795 thru 0x2b7056d (0xcdd9 bytes) 2024-09-16T13:17:04.250 INFO:teuthology.orchestra.run.smithi098.stdout:683 trunc from 0xc55a95d to 0x74820bd 2024-09-16T13:17:04.272 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.280 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:04.280 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:04.281 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:04.288 INFO:teuthology.orchestra.run.smithi098.stdout:686 write 0xac14cab thru 0xac14f91 (0x2e7 bytes) 2024-09-16T13:17:04.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:04.295 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.295 INFO:teuthology.orchestra.run.smithi098.stdout:688 read 0x9f0efd1 thru 0x9f13049 (0x4079 bytes) 2024-09-16T13:17:04.309 INFO:teuthology.orchestra.run.smithi098.stdout:689 write 0xbd59970 thru 0xbd6463f (0xacd0 bytes) 2024-09-16T13:17:04.318 INFO:teuthology.orchestra.run.smithi098.stdout:690 read 0x770c2da thru 0x7711101 (0x4e28 bytes) 2024-09-16T13:17:04.318 INFO:teuthology.orchestra.run.smithi098.stdout:691 writesame 0x15dd8d3 thru 0x15e9318 data_size 0x2(0xba46 bytes) 2024-09-16T13:17:04.333 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:04.328+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:04.343 INFO:teuthology.orchestra.run.smithi098.stdout:692 write 0xcfe3e thru 0xd6df1 (0x6fb4 bytes) 2024-09-16T13:17:04.352 INFO:teuthology.orchestra.run.smithi098.stdout:695 write 0x26cc565 thru 0x26d824b (0xbce7 bytes) 2024-09-16T13:17:04.360 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.374 INFO:teuthology.orchestra.run.smithi098.stdout:697 trunc from 0xbd64640 to 0x5787a92 2024-09-16T13:17:04.400 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:04.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:830 3.9 e27/23 3:940847c8:::rbd_data.112de98401a1.000000000000001b:head v 27'6, mlcod=27'4)", 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.244050+0000", 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.1495510500000004, 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017733099999999999, 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244050+0000", 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244050+0000", 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244052+0000", 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9879999999999999e-06 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244158+0000", 2024-09-16T13:17:04.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105241 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244190+0000", 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2484999999999998e-05 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244202+0000", 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1722999999999999e-05 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244295+0000", 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3566999999999994e-05 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244326+0000", 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0849000000000002e-05 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.245722+0000", 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001396331 2024-09-16T13:17:04.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.245824+0000", 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010112599999999999 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:906 3.0 e27/23 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head v 27'7, mlcod=27'5)", 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.428930+0000", 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.9646709399999995, 2024-09-16T13:17:04.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020851440000000001, 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428930+0000", 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428930+0000", 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428932+0000", 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9570000000000001e-06 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.429019+0000", 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6584999999999999e-05 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.429055+0000", 2024-09-16T13:17:04.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5573e-05 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.429068+0000", 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3362e-05 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.429196+0000", 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000128558 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.429260+0000", 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3328000000000001e-05 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.430924+0000", 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001664366 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.431016+0000", 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1415000000000005e-05 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:989 3.0 e27/23 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head v 27'8, mlcod=27'6)", 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.647434+0000", 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.7461671540000001, 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021370790000000001, 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647434+0000", 2024-09-16T13:17:04.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647434+0000", 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647436+0000", 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7799999999999999e-06 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647502+0000", 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6135000000000003e-05 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647535+0000", 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3315999999999997e-05 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647547+0000", 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1866e-05 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647685+0000", 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013769699999999999 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647739+0000", 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3699000000000002e-05 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.649487+0000", 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017479240000000001 2024-09-16T13:17:04.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.649571+0000", 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4661999999999997e-05 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:990 3.0 e27/23 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head v 27'9, mlcod=27'7)", 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.651339+0000", 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.7422621679999999, 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018613659999999999, 2024-09-16T13:17:04.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651339+0000", 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651339+0000", 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651340+0000", 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.297e-06 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651367+0000", 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6376e-05 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651392+0000", 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.472e-05 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651402+0000", 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0232e-05 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651513+0000", 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000111535 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651584+0000", 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0204999999999996e-05 2024-09-16T13:17:04.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.653130+0000", 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015469749999999999 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.653201+0000", 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0025999999999997e-05 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:992 3.0 e27/23 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head v 27'10, mlcod=27'8)", 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.660178+0000", 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.7334234779999997, 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015958999999999999, 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.660178+0000", 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.660178+0000", 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.660180+0000", 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0430000000000002e-06 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.660199+0000", 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9332999999999999e-05 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.660242+0000", 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2865999999999999e-05 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.660260+0000", 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7805999999999999e-05 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.660384+0000", 2024-09-16T13:17:04.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123907 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.660432+0000", 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7822999999999999e-05 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.661729+0000", 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012969769999999999 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.661774+0000", 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5145000000000002e-05 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1031 3.4 e27/23 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head v 27'3, mlcod=26'1)", 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.777094+0000", 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6165071510000004, 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016652240000000001, 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.777094+0000", 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.777094+0000", 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.777096+0000", 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.863e-06 2024-09-16T13:17:04.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.777127+0000", 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0496000000000001e-05 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.777160+0000", 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3448000000000002e-05 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.777172+0000", 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1908e-05 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.777273+0000", 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101328 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.777318+0000", 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4626999999999998e-05 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.778696+0000", 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001377805 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.778759+0000", 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3749000000000002e-05 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1264 3.4 e27/23 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head v 27'4, mlcod=26'2)", 2024-09-16T13:17:04.427 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.8923525540000004, 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026410209999999999, 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:04.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501250+0000", 2024-09-16T13:17:04.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2410000000000001e-06 2024-09-16T13:17:04.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501259+0000", 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501277+0000", 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8683999999999999e-05 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501288+0000", 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.065e-05 2024-09-16T13:17:04.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501562+0000", 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027421200000000002 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501618+0000", 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6045000000000003e-05 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503841+0000", 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022230829999999998 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503890+0000", 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8526000000000003e-05 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.435 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1263 3.0 e27/23 3:0214ce37:::rbd_data.112de98401a1.0000000000000007:head v 27'12, mlcod=27'10)", 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.501307+0000", 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.8922939679999997, 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018975210000000001, 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501307+0000", 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501307+0000", 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501308+0000", 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1199999999999998e-07 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501316+0000", 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1599999999999998e-06 2024-09-16T13:17:04.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501323+0000", 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.233e-06 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501328+0000", 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7099999999999998e-06 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501618+0000", 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028967799999999997 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501671+0000", 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3869999999999998e-05 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503092+0000", 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014201859999999999 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503205+0000", 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113272 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1906 3.9 e31/23 3:90f1252a:::rbd_data.112d2ce070e7.000000000000002e:head v 31'10, mlcod=27'7)", 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.015221+0000", 2024-09-16T13:17:04.440 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.378380428, 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001825651, 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015221+0000", 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015221+0000", 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015221+0000", 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.28e-07 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015225+0000", 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8199999999999998e-06 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015233+0000", 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6690000000000006e-06 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015239+0000", 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2580000000000001e-06 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015316+0000", 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7392999999999996e-05 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015341+0000", 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4258000000000001e-05 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.017010+0000", 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016689999999999999 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.017047+0000", 2024-09-16T13:17:04.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7024999999999999e-05 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2361 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'23, mlcod=31'21)", 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.864739+0000", 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.52886198299999998, 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019981550000000002, 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864739+0000", 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864739+0000", 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864741+0000", 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.866e-06 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864790+0000", 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9199000000000001e-05 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864836+0000", 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5108999999999998e-05 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864848+0000", 2024-09-16T13:17:04.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2193e-05 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864972+0000", 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000124024 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.865047+0000", 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5074999999999995e-05 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.866622+0000", 2024-09-16T13:17:04.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001575629 2024-09-16T13:17:04.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.866737+0000", 2024-09-16T13:17:04.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011506 2024-09-16T13:17:04.449 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.449 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.449 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.520 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2364 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'14, mlcod=31'12)", 2024-09-16T13:17:04.520 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.872224+0000", 2024-09-16T13:17:04.520 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.52137699000000004, 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016172179999999999, 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.872224+0000", 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.872224+0000", 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.872226+0000", 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9139999999999998e-06 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.872252+0000", 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5738999999999999e-05 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.872285+0000", 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3312000000000002e-05 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.872297+0000", 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.199e-05 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.872366+0000", 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8752999999999997e-05 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.872392+0000", 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.561e-05 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.873797+0000", 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014054110000000001 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.873842+0000", 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4489e-05 2024-09-16T13:17:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2391 3.0 e31/23 3:01a02ab3:::rbd_data.112dbfd5d823.000000000000000c:head v 31'24, mlcod=31'22)", 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.898493+0000", 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.49510809300000003, 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001790378, 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898493+0000", 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898493+0000", 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898495+0000", 2024-09-16T13:17:04.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.165e-06 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898522+0000", 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7010999999999999e-05 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898558+0000", 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5476999999999999e-05 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898571+0000", 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3478999999999999e-05 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898680+0000", 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109064 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898732+0000", 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1385999999999999e-05 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.900202+0000", 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001469852 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.900284+0000", 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1944e-05 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2428 3.9 e31/23 3:95ed565e:::rbd_data.112dbfd5d823.0000000000000030:head v 31'16, mlcod=31'14)", 2024-09-16T13:17:04.529 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.005549+0000", 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.38805237199999998, 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016176350000000001, 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.005549+0000", 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.005549+0000", 2024-09-16T13:17:04.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.005550+0000", 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.268e-06 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.005563+0000", 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2519e-05 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.005587+0000", 2024-09-16T13:17:04.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4572000000000001e-05 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.005597+0000", 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3789999999999995e-06 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.005730+0000", 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013296799999999999 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.005791+0000", 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1575999999999995e-05 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007107+0000", 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001316188 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007167+0000", 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9165e-05 2024-09-16T13:17:04.533 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.38620886599999998, 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:17:04.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.38542336700000002, 2024-09-16T13:17:04.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:17:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.384256445, 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:17:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:17:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.38138181399999999, 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.378368074, 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:17:04.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:17:04.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.35316387100000002, 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2623 3.1f e31/30 3:ffd0fa40:::rbd_data.112dbfd5d823.0000000000000328:head v 31'14, mlcod=31'12)", 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.184023+0000", 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.209578452, 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001717302, 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:04.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184023+0000", 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184023+0000", 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184025+0000", 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:04.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184047+0000", 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2313000000000001e-05 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184088+0000", 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0754999999999999e-05 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184107+0000", 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8957000000000002e-05 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184242+0000", 2024-09-16T13:17:04.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000135248 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184288+0000", 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5945000000000001e-05 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.185652+0000", 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001363471 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.185740+0000", 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8332999999999996e-05 2024-09-16T13:17:04.563 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.564 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.564 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.564 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:04.564 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:04.564 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:04.565 INFO:teuthology.orchestra.run.smithi098.stdout:698 trunc from 0x5787a92 to 0xa45b5c0 2024-09-16T13:17:04.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:04.567 INFO:teuthology.orchestra.run.smithi098.stdout:699 punch from 0x485b4e5 to 0x4866d95, (0xb8b0 bytes) 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:700 writesame 0xb4f1528 thru 0xb4f2263 data_size 0xb(0xd3c bytes) 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:701 punch from 0x67ab6c6 to 0x67b9005, (0xd93f bytes) 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:703 read 0x338e3be thru 0x33991a7 (0xadea bytes) 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:704 write 0x58019ca thru 0x580c553 (0xab8a bytes) 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:705 write 0xb858805 thru 0xb868539 (0xfd35 bytes) 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:706 read 0x129a2fd thru 0x12a1374 (0x7078 bytes) 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:707 write 0x81ff9ca thru 0x8207111 (0x7748 bytes) 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:708 trunc from 0xb86853a to 0xb95e7de 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:709 trunc from 0xb95e7de to 0x67024e1 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:712 read 0x6270549 thru 0x627fecd (0xf985 bytes) 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:714 writesame 0xbbd5fd6 thru 0xbbe1482 data_size 0x1(0xb4ad bytes) 2024-09-16T13:17:04.568 INFO:teuthology.orchestra.run.smithi098.stdout:715 punch from 0x5fba90f to 0x5fc5b76, (0xb267 bytes) 2024-09-16T13:17:04.569 INFO:teuthology.orchestra.run.smithi098.stdout:716 punch from 0xb07d97b to 0xb08788f, (0x9f14 bytes) 2024-09-16T13:17:04.569 INFO:teuthology.orchestra.run.smithi098.stdout:717 write 0xdac2716 thru 0xdacab1e (0x8409 bytes) 2024-09-16T13:17:04.569 INFO:teuthology.orchestra.run.smithi098.stdout:719 writesame 0x107fb1c thru 0x1083571 data_size 0x6(0x3a56 bytes) 2024-09-16T13:17:04.569 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:04.560+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:04.587 INFO:teuthology.orchestra.run.smithi098.stdout:720 write 0xe2e550b thru 0xe2e7058 (0x1b4e bytes) 2024-09-16T13:17:04.594 INFO:teuthology.orchestra.run.smithi098.stdout:722 write 0xcee51a9 thru 0xcee6a3b (0x1893 bytes) 2024-09-16T13:17:04.600 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.600 INFO:teuthology.orchestra.run.smithi098.stdout:724 writesame 0xa6f1c0b thru 0xa6f6bd4 data_size 0x7(0x4fca bytes) 2024-09-16T13:17:04.636 INFO:teuthology.orchestra.run.smithi098.stdout:725 trunc from 0xe2e7059 to 0x103b99a 2024-09-16T13:17:04.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:04.688 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:04.684+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:04.691 INFO:teuthology.orchestra.run.smithi098.stdout:726 write 0x9e8c216 thru 0x9e930a5 (0x6e90 bytes) 2024-09-16T13:17:04.699 INFO:teuthology.orchestra.run.smithi098.stdout:727 punch from 0xc18acd to 0xc25961, (0xce94 bytes) 2024-09-16T13:17:04.699 INFO:teuthology.orchestra.run.smithi098.stdout:729 write 0xa669a6 thru 0xa75c36 (0xf291 bytes) 2024-09-16T13:17:04.706 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.707 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:04.707 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:04.707 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:04.708 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:04.716 INFO:teuthology.orchestra.run.smithi098.stdout:731 write 0xa791cca thru 0xa795f8d (0x42c4 bytes) 2024-09-16T13:17:04.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:04.722 INFO:teuthology.orchestra.run.smithi098.stdout:732 punch from 0x44edbb to 0x459f47, (0xb18c bytes) 2024-09-16T13:17:04.722 INFO:teuthology.orchestra.run.smithi098.stdout:733 write 0x315763e thru 0x31605b3 (0x8f76 bytes) 2024-09-16T13:17:04.741 INFO:teuthology.orchestra.run.smithi098.stdout:734 writesame 0xb9ec338 thru 0xb9ef4db data_size 0x12(0x31a4 bytes) 2024-09-16T13:17:04.741 INFO:teuthology.orchestra.run.smithi098.stdout:735 write 0x9ada700 thru 0x9ae7b22 (0xd423 bytes) 2024-09-16T13:17:04.749 INFO:teuthology.orchestra.run.smithi098.stdout:736 writesame 0xa536957 thru 0xa53edcc data_size 0x5(0x8476 bytes) 2024-09-16T13:17:04.759 INFO:teuthology.orchestra.run.smithi098.stdout:737 writesame 0x7c7856f thru 0x7c87b4a data_size 0x1e(0xf5dc bytes) 2024-09-16T13:17:04.767 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:04.762+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:04.768 INFO:teuthology.orchestra.run.smithi098.stdout:738 read 0x31638c6 thru 0x3169c51 (0x638c bytes) 2024-09-16T13:17:04.768 INFO:teuthology.orchestra.run.smithi098.stdout:739 read 0x15e5be0 thru 0x15ed15b (0x757c bytes) 2024-09-16T13:17:04.768 INFO:teuthology.orchestra.run.smithi098.stdout:740 write 0xb584ef thru 0xb5c398 (0x3eaa bytes) 2024-09-16T13:17:04.773 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.773 INFO:teuthology.orchestra.run.smithi098.stdout:742 write 0x3ed9209 thru 0x3ee14b7 (0x82af bytes) 2024-09-16T13:17:04.780 INFO:teuthology.orchestra.run.smithi098.stdout:744 writesame 0xb66113 thru 0xb67fc2 data_size 0x8(0x1eb0 bytes) 2024-09-16T13:17:04.786 INFO:teuthology.orchestra.run.smithi098.stdout:749 read 0xb707da4 thru 0xb70a11e (0x237b bytes) 2024-09-16T13:17:04.786 INFO:teuthology.orchestra.run.smithi098.stdout:750 write 0x3e4a72e thru 0x3e541fd (0x9ad0 bytes) 2024-09-16T13:17:04.786 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:04.793 INFO:teuthology.orchestra.run.smithi098.stdout:751 read 0x25fd7ab thru 0x260c19e (0xe9f4 bytes) 2024-09-16T13:17:04.793 INFO:teuthology.orchestra.run.smithi098.stdout:752 punch from 0x678a850 to 0x679a7bb, (0xff6b bytes) 2024-09-16T13:17:04.805 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:04.805 INFO:teuthology.orchestra.run.smithi098.stdout:753 trunc from 0xb9ef4dc to 0x4e1e42c 2024-09-16T13:17:04.827 INFO:teuthology.orchestra.run.smithi098.stdout:754 read 0x1b59cf4 thru 0x1b5db14 (0x3e21 bytes) 2024-09-16T13:17:04.833 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:04.833 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:04.833 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:04.833 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:04.834 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:04.836 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:04.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:04.852 INFO:teuthology.orchestra.run.smithi098.stdout:755 write 0xd7e735c thru 0xd7f0b67 (0x980c bytes) 2024-09-16T13:17:04.858 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:04.864 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.864 INFO:teuthology.orchestra.run.smithi098.stdout:truncating to largest ever: 0xee02890 2024-09-16T13:17:04.866 INFO:teuthology.orchestra.run.smithi098.stdout:758 trunc from 0xd7f0b68 to 0xee02890 2024-09-16T13:17:04.868 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:04.864+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:04.876 INFO:teuthology.orchestra.run.smithi098.stdout:759 writesame 0xa95e25d thru 0xa96566c data_size 0x18(0x7410 bytes) 2024-09-16T13:17:04.880 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:04.886 INFO:teuthology.orchestra.run.smithi098.stdout:760 writesame 0x7264ea1 thru 0x72694c0 data_size 0x58(0x4620 bytes) 2024-09-16T13:17:04.892 INFO:teuthology.orchestra.run.smithi098.stdout:761 write 0xb07b4e5 thru 0xb0861f1 (0xad0d bytes) 2024-09-16T13:17:04.902 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:04.903 INFO:teuthology.orchestra.run.smithi098.stdout:762 write 0x939217e thru 0x939d874 (0xb6f7 bytes) 2024-09-16T13:17:04.908 INFO:teuthology.orchestra.run.smithi098.stdout:763 writesame 0x2b3c800 thru 0x2b4ae00 data_size 0x13(0xe601 bytes) 2024-09-16T13:17:04.918 INFO:teuthology.orchestra.run.smithi098.stdout:764 write 0xe16cab8 thru 0xe16cd27 (0x270 bytes) 2024-09-16T13:17:04.923 INFO:teuthology.orchestra.run.smithi098.stdout:765 write 0xd8f217c thru 0xd8f3f3d (0x1dc2 bytes) 2024-09-16T13:17:04.925 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:04.930 INFO:teuthology.orchestra.run.smithi098.stdout:766 read 0xd3e0b81 thru 0xd3e500c (0x448c bytes) 2024-09-16T13:17:04.930 INFO:teuthology.orchestra.run.smithi098.stdout:768 read 0xa84360b thru 0xa84db4b (0xa541 bytes) 2024-09-16T13:17:04.930 INFO:teuthology.orchestra.run.smithi098.stdout:771 read 0x8dd7229 thru 0x8ddaaa6 (0x387e bytes) 2024-09-16T13:17:04.930 INFO:teuthology.orchestra.run.smithi098.stdout:772 write 0xc09e55d thru 0xc09e786 (0x22a bytes) 2024-09-16T13:17:04.935 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.935 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:04.935 INFO:teuthology.orchestra.run.smithi098.stdout:775 read 0xe1cb1e3 thru 0xe1cb9ba (0x7d8 bytes) 2024-09-16T13:17:04.935 INFO:teuthology.orchestra.run.smithi098.stdout:776 punch from 0x1710e54 to 0x1712e1e, (0x1fca bytes) 2024-09-16T13:17:04.944 INFO:teuthology.orchestra.run.smithi098.stdout:778 trunc from 0xee02890 to 0xa9f33af 2024-09-16T13:17:04.945 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:04.945 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:04.946 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:04.948 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:04.949 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:04.949 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:04.949 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:04.949 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:04.949 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:04.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:04.974 INFO:teuthology.orchestra.run.smithi098.stdout:779 write 0xadbadc5 thru 0xadbccc0 (0x1efc bytes) 2024-09-16T13:17:04.979 INFO:teuthology.orchestra.run.smithi098.stdout:780 clone 3 order 19 su 131072 sc 8 2024-09-16T13:17:05.002 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:04.998+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:05.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:17:05.057 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:05.057 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:05.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:05.069 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:05.062+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:05.147 INFO:teuthology.orchestra.run.smithi098.stdout:flattening image image_client.0-clone2 2024-09-16T13:17:05.170 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:05.164+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:05.175 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:05.175 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:05.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:05.271 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:05.266+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:05.276 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:05.276 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:05.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:05.353 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #1, image image_client.0-clone1 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent2 2024-09-16T13:17:05.373 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:05.368+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:05.439 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:05.439 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:05.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:05.474 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:05.470+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:05.504 INFO:teuthology.orchestra.run.smithi098.stdout:782 write 0xd9c7a01 thru 0xd9cdb0b (0x610b bytes) 2024-09-16T13:17:05.535 INFO:teuthology.orchestra.run.smithi098.stdout:783 trunc from 0xd9cdb0c to 0x69db96 2024-09-16T13:17:05.546 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:05.546 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:05.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:05.575 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:05.568+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:05.576 INFO:teuthology.orchestra.run.smithi098.stdout:784 write 0x962cce thru 0x970000 (0xd333 bytes) 2024-09-16T13:17:05.606 INFO:teuthology.orchestra.run.smithi098.stdout:785 write 0xa22f6cb thru 0xa230d74 (0x16aa bytes) 2024-09-16T13:17:05.630 INFO:teuthology.orchestra.run.smithi098.stdout:788 writesame 0xeac57d1 thru 0xeac8b0f data_size 0x1(0x333f bytes) 2024-09-16T13:17:05.630 INFO:teuthology.orchestra.run.smithi098.stdout:795 read 0xdae7dc thru 0xdb5d5a (0x757f bytes) 2024-09-16T13:17:05.630 INFO:teuthology.orchestra.run.smithi098.stdout:796 write 0x23f1c thru 0x27569 (0x364e bytes) 2024-09-16T13:17:05.636 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:05.636 INFO:teuthology.orchestra.run.smithi098.stdout:799 write 0x8507339 thru 0x850f625 (0x82ed bytes) 2024-09-16T13:17:05.642 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:05.642 INFO:teuthology.orchestra.run.smithi098.stdout:801 write 0x6c5ac6e thru 0x6c61b54 (0x6ee7 bytes) 2024-09-16T13:17:05.649 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:05.649 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:05.649 INFO:teuthology.orchestra.run.smithi098.stdout:802 write 0xe1e4aea thru 0xe1e641b (0x1932 bytes) 2024-09-16T13:17:05.655 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:05.655 INFO:teuthology.orchestra.run.smithi098.stdout:804 read 0x7eeead2 thru 0x7efd1d1 (0xe700 bytes) 2024-09-16T13:17:05.655 INFO:teuthology.orchestra.run.smithi098.stdout:805 read 0x6cf92de thru 0x6cff223 (0x5f46 bytes) 2024-09-16T13:17:05.655 INFO:teuthology.orchestra.run.smithi098.stdout:806 read 0x9a2561d thru 0x9a2e50f (0x8ef3 bytes) 2024-09-16T13:17:05.656 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:05.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:05.674 INFO:teuthology.orchestra.run.smithi098.stdout:808 trunc from 0xeac8b10 to 0x550de4a 2024-09-16T13:17:05.676 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:05.672+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:05.695 INFO:teuthology.orchestra.run.smithi098.stdout:809 write 0x49de15f thru 0x49e8036 (0x9ed8 bytes) 2024-09-16T13:17:05.715 INFO:teuthology.orchestra.run.smithi098.stdout:810 write 0x5dd794a thru 0x5de1a00 (0xa0b7 bytes) 2024-09-16T13:17:05.748 INFO:teuthology.orchestra.run.smithi098.stdout:811 writesame 0xb580117 thru 0xb584ea2 data_size 0xe(0x4d8c bytes) 2024-09-16T13:17:05.748 INFO:teuthology.orchestra.run.smithi098.stdout:812 write 0x4b91662 thru 0x4b9dbd6 (0xc575 bytes) 2024-09-16T13:17:05.767 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:05.767 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:05.769 INFO:teuthology.orchestra.run.smithi098.stdout:814 write 0xcfcf757 thru 0xcfdea15 (0xf2bf bytes) 2024-09-16T13:17:05.775 INFO:teuthology.orchestra.run.smithi098.stdout:815 punch from 0x41aaac to 0x4201c1, (0x5715 bytes) 2024-09-16T13:17:05.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:05.778 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:05.772+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:05.779 INFO:teuthology.orchestra.run.smithi098.stdout:816 trunc from 0xcfdea16 to 0xe9f716e 2024-09-16T13:17:05.790 INFO:teuthology.orchestra.run.smithi098.stdout:817 read 0xc1309b8 thru 0xc132bd0 (0x2219 bytes) 2024-09-16T13:17:05.790 INFO:teuthology.orchestra.run.smithi098.stdout:818 writesame 0x35ab1d2 thru 0x35b6617 data_size 0x32(0xb446 bytes) 2024-09-16T13:17:05.799 INFO:teuthology.orchestra.run.smithi098.stdout:819 writesame 0xa72716e thru 0xa72aee8 data_size 0x1(0x3d7b bytes) 2024-09-16T13:17:05.819 INFO:teuthology.orchestra.run.smithi098.stdout:820 punch from 0xa9766c1 to 0xa97d8ee, (0x722d bytes) 2024-09-16T13:17:05.838 INFO:teuthology.orchestra.run.smithi098.stdout:822 trunc from 0xe9f716e to 0x57c8c3c 2024-09-16T13:17:05.861 INFO:teuthology.orchestra.run.smithi098.stdout:823 writesame 0x2697d6e thru 0x26a5cbb data_size 0x65(0xdf4e bytes) 2024-09-16T13:17:05.876 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:05.876 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:05.878 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:05.870+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:05.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:05.892 INFO:teuthology.orchestra.run.smithi098.stdout:825 write 0xd14fc7f thru 0xd158c20 (0x8fa2 bytes) 2024-09-16T13:17:05.897 INFO:teuthology.orchestra.run.smithi098.stdout:826 writesame 0xc696184 thru 0xc6a1ef4 data_size 0x1(0xbd71 bytes) 2024-09-16T13:17:05.947 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:05.948 INFO:teuthology.orchestra.run.smithi098.stdout:828 read 0x1f13e39 thru 0x1f1cc04 (0x8dcc bytes) 2024-09-16T13:17:05.948 INFO:teuthology.orchestra.run.smithi098.stdout:831 read 0x7d4b22d thru 0x7d52795 (0x7569 bytes) 2024-09-16T13:17:05.948 INFO:teuthology.orchestra.run.smithi098.stdout:832 punch from 0x7dec1bb to 0x7df37d0, (0x7615 bytes) 2024-09-16T13:17:05.963 INFO:teuthology.orchestra.run.smithi098.stdout:834 trunc from 0xd158c21 to 0x51b2896 2024-09-16T13:17:05.979 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:05.972+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:05.989 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:05.990 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:06.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:06.002 INFO:teuthology.orchestra.run.smithi098.stdout:835 write 0x65bc0d9 thru 0x65bfca5 (0x3bcd bytes) 2024-09-16T13:17:06.009 INFO:teuthology.orchestra.run.smithi098.stdout:836 read 0x14459b7 thru 0x14481a1 (0x27eb bytes) 2024-09-16T13:17:06.009 INFO:teuthology.orchestra.run.smithi098.stdout:837 punch from 0x6de0d4 to 0x6e0866, (0x2792 bytes) 2024-09-16T13:17:06.009 INFO:teuthology.orchestra.run.smithi098.stdout:838 read 0x8f0df7 thru 0x8f695a (0x5b64 bytes) 2024-09-16T13:17:06.010 INFO:teuthology.orchestra.run.smithi098.stdout:840 write 0x24d51cb thru 0x24e0124 (0xaf5a bytes) 2024-09-16T13:17:06.020 INFO:teuthology.orchestra.run.smithi098.stdout:841 punch from 0x16d9b50 to 0x16e1d5e, (0x820e bytes) 2024-09-16T13:17:06.020 INFO:teuthology.orchestra.run.smithi098.stdout:842 punch from 0x59db54c to 0x59dd7a6, (0x225a bytes) 2024-09-16T13:17:06.026 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:17:06.038 INFO:teuthology.orchestra.run.smithi098.stdout:843 write 0x8aafebf thru 0x8ab76d2 (0x7814 bytes) 2024-09-16T13:17:06.046 INFO:teuthology.orchestra.run.smithi098.stdout:845 trunc from 0x8ab76d3 to 0x9dad282 2024-09-16T13:17:06.069 INFO:teuthology.orchestra.run.smithi098.stdout:846 writesame 0xbd0987c thru 0xbd0cf17 data_size 0xc(0x369c bytes) 2024-09-16T13:17:06.069 INFO:teuthology.orchestra.run.smithi098.stdout:848 writesame 0x74c9b09 thru 0x74cfb25 data_size 0x25(0x601d bytes) 2024-09-16T13:17:06.080 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:06.074+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:06.081 INFO:teuthology.orchestra.run.smithi098.stdout:849 trunc from 0xbd0cf18 to 0x9a9a7a7 2024-09-16T13:17:06.098 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:06.098 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:06.099 INFO:teuthology.orchestra.run.smithi098.stdout:851 trunc from 0x9a9a7a7 to 0xdeb7a53 2024-09-16T13:17:06.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:06.110 INFO:teuthology.orchestra.run.smithi098.stdout:852 write 0x6f9a621 thru 0x6f9d410 (0x2df0 bytes) 2024-09-16T13:17:06.116 INFO:teuthology.orchestra.run.smithi098.stdout:853 read 0x5329e15 thru 0x533764b (0xd837 bytes) 2024-09-16T13:17:06.116 INFO:teuthology.orchestra.run.smithi098.stdout:854 write 0x9541c7c thru 0x9543836 (0x1bbb bytes) 2024-09-16T13:17:06.126 INFO:teuthology.orchestra.run.smithi098.stdout:855 read 0xb9991ad thru 0xb99fc41 (0x6a95 bytes) 2024-09-16T13:17:06.126 INFO:teuthology.orchestra.run.smithi098.stdout:856 write 0x535e536 thru 0x536a5d8 (0xc0a3 bytes) 2024-09-16T13:17:06.133 INFO:teuthology.orchestra.run.smithi098.stdout:857 write 0x133284c thru 0x133a3ff (0x7bb4 bytes) 2024-09-16T13:17:06.138 INFO:teuthology.orchestra.run.smithi098.stdout:858 read 0x6c8d58a thru 0x6c99e24 (0xc89b bytes) 2024-09-16T13:17:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:859 write 0xa57b5e4 thru 0xa583e23 (0x8840 bytes) 2024-09-16T13:17:06.146 INFO:teuthology.orchestra.run.smithi098.stdout:861 read 0xd5a094f thru 0xd5a925b (0x890d bytes) 2024-09-16T13:17:06.146 INFO:teuthology.orchestra.run.smithi098.stdout:862 write 0x5d2352d thru 0x5d30d5b (0xd82f bytes) 2024-09-16T13:17:06.152 INFO:teuthology.orchestra.run.smithi098.stdout:863 write 0x93bc50d thru 0x93c5221 (0x8d15 bytes) 2024-09-16T13:17:06.159 INFO:teuthology.orchestra.run.smithi098.stdout:865 writesame 0x6fb1eaa thru 0x6fb51d0 data_size 0x1b(0x3327 bytes) 2024-09-16T13:17:06.166 INFO:teuthology.orchestra.run.smithi098.stdout:866 read 0xbc9b0d thru 0xbd03b1 (0x68a5 bytes) 2024-09-16T13:17:06.166 INFO:teuthology.orchestra.run.smithi098.stdout:868 clone 4 order 18 su 32768 sc 15 2024-09-16T13:17:06.180 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:06.176+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:06.206 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:06.207 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:06.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:06.281 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:06.276+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:06.311 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:06.311 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:06.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:06.382 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:06.376+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:06.431 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:06.431 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:06.441 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:06.483 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:06.478+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:06.543 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:06.543 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:06.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:06.584 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:06.576+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:06.651 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:06.652 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:06.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:06.705 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:06.700+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:06.786 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:06.782+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:06.808 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:06.808 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:06.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:06.887 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:06.882+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:06.913 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:06.913 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:06.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:06.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:06.987 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:06.982+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:07.017 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:07.017 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:07.019 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:07.019 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:07.019 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T13:17:07.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T13:17:07.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:07.089 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:07.082+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:07.100 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:07.122 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:07.130 INFO:teuthology.orchestra.run.smithi098.stdout:flattening image image_client.0-clone3 2024-09-16T13:17:07.151 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:07.169 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:07.189 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:07.182+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:07.189 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:07.190 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:07.190 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:07.190 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:07.191 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:07.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:07.213 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:07.235 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:07.257 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:07.260 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:07.260 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:07.260 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:07.260 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:07.260 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:07.260 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:07.260 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:07.261 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:07.290 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:07.284+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:07.329 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:07.329 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:07.329 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:07.329 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:07.329 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:07.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:07.363 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #2, image image_client.0-clone2 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent3 2024-09-16T13:17:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:17:07.373 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 34, 2024-09-16T13:17:07.373 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "6ddcae89-6c18-4eb7-8712-1c53f3b07278", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-16T13:16:23.176689+0000", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-16T13:17:07.012782+0000", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "last_up_change": "2024-09-16T13:16:33.765413+0000", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "last_in_change": "2024-09-16T13:16:25.201328+0000", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_num": 5806080, 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_set": [ 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "nodeep-scrub", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "noscrub", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "pglog_hardlimit", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "purged_snapdirs", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_deletes", 2024-09-16T13:17:07.374 INFO:teuthology.orchestra.run.smithi098.stdout: "sortbitwise" 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_version": 5, 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_snapshot": "", 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_max": 3, 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd": 8, 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "require_min_compat_client": "luminous", 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min_compat_client": "jewel", 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "require_osd_release": "quincy", 2024-09-16T13:17:07.375 INFO:teuthology.orchestra.run.smithi098.stdout: "pools": [ 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 1, 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": ".mgr", 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:33.497561+0000", 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool", 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:17:07.376 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 1, 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 1, 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 1, 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 1, 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 1, 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:17:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "17", 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 0, 2024-09-16T13:17:07.378 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 0, 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:17:07.379 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:17:07.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "options": { 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_max": 32, 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_min": 1 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:17:07.381 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": {} 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 2, 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": "rbd", 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:40.994538+0000", 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 8193, 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:17:07.382 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 8, 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 8, 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 8, 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 8, 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 8, 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:17:07.383 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "21", 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:17:07.384 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 3, 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 21, 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:17:07.385 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:17:07.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: "options": {}, 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd": {} 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 3, 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": "pool_client.0", 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:50.759705+0000", 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 8193, 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:17:07.388 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 32, 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 32, 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 32, 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 32, 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 32, 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:17:07.389 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "34", 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 7, 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 34, 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:17:07.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:17:07.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "options": {}, 2024-09-16T13:17:07.392 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:17:07.393 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd": {} 2024-09-16T13:17:07.393 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.393 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.393 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:07.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osds": [ 2024-09-16T13:17:07.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 0, 2024-09-16T13:17:07.393 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "be8543b4-3d28-4b37-a4ce-d2a56a09aeaf", 2024-09-16T13:17:07.393 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:07.393 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0.8320770263671875, 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6808", 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6809", 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6810", 2024-09-16T13:17:07.395 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6811", 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6816", 2024-09-16T13:17:07.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:07.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6817", 2024-09-16T13:17:07.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:07.397 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.397 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6813", 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6814", 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:07.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6809/22271", 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6811/22271", 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6817/22271", 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6814/22271", 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 1, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "a7f95c7b-56d4-437b-a160-cabb8fd2b1e4", 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6812", 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6815", 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6818", 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6819", 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6822", 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6823", 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6820", 2024-09-16T13:17:07.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:07.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6821", 2024-09-16T13:17:07.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:07.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6815/22272", 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6819/22272", 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6823/22272", 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6821/22272", 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 2, 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c", 2024-09-16T13:17:07.405 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 18, 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6800", 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6801", 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6802", 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6803", 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6806", 2024-09-16T13:17:07.409 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:07.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6807", 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6804", 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6805", 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6801/22273", 2024-09-16T13:17:07.411 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6803/22273", 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6807/22273", 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6805/22273", 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 3, 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "4898b5b5-b912-430c-adcb-c52d90226917", 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:07.412 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 30, 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6824", 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6825", 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6826", 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:07.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6827", 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6830", 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6831", 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6828", 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6829", 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6825/22270", 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6827/22270", 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6831/22270", 2024-09-16T13:17:07.417 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6829/22270", 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 4, 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "6c478be2-f606-41d8-967c-9a689f09a440", 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:07.418 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 30, 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6824", 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6825", 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:07.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6826", 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6827", 2024-09-16T13:17:07.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6830", 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6831", 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6828", 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6829", 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6825/22286", 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6827/22286", 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6831/22286", 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6829/22286", 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:07.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 5, 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "5001c828-b223-4d2c-b893-f9c56da73b83", 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 30, 2024-09-16T13:17:07.424 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6808", 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6809", 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:07.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6810", 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6811", 2024-09-16T13:17:07.426 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6814", 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6815", 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6812", 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6813", 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6809/22285", 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6811/22285", 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6815/22285", 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6813/22285", 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 6, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "6f7a9125-674c-4be2-93e1-b6296cd00f8e", 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:07.430 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6816", 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6817", 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.431 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6818", 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6819", 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.432 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6822", 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6823", 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6820", 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6821", 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6817/22315", 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6819/22315", 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6823/22315", 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6821/22315", 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 7, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "bcdfe435-f24d-45a9-a629-7380ad2e575a", 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:07.436 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6800", 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6801", 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:07.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6802", 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6803", 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6806", 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6807", 2024-09-16T13:17:07.439 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6804", 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6805", 2024-09-16T13:17:07.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6801/22284", 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6803/22284", 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6807/22284", 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6805/22284", 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_xinfo": [ 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 0, 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.191311+0000", 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 1, 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:07.442 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.563551+0000", 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 2, 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:07.443 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.493457+0000", 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 3, 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.599815+0000", 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.444 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 4, 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.634352+0000", 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 5, 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:07.445 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.262873+0000", 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 6, 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.508881+0000", 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 7, 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.348276+0000", 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_upmap": [], 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_upmap_items": [], 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_temp": [], 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_temp": [], 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklist": {}, 2024-09-16T13:17:07.447 INFO:teuthology.orchestra.run.smithi098.stdout: "range_blocklist": {}, 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profiles": { 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "default": { 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-failure-domain": "osd", 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "k": "2", 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "m": "1", 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "plugin": "jerasure", 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "technique": "reed_sol_van" 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps_queue": [], 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "new_removed_snaps": [], 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "new_purged_snaps": [], 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_node_flags": {}, 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "device_class_flags": {}, 2024-09-16T13:17:07.448 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": { 2024-09-16T13:17:07.449 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode_enabled": false, 2024-09-16T13:17:07.449 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_bucket_count": 0, 2024-09-16T13:17:07.449 INFO:teuthology.orchestra.run.smithi098.stdout: "degraded_stretch_mode": 0, 2024-09-16T13:17:07.449 INFO:teuthology.orchestra.run.smithi098.stdout: "recovering_stretch_mode": 0, 2024-09-16T13:17:07.449 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode_bucket": 0 2024-09-16T13:17:07.449 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.449 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:07.451 INFO:tasks.thrashosds.thrasher:j is {'epoch': 34, 'fsid': '6ddcae89-6c18-4eb7-8712-1c53f3b07278', 'created': '2024-09-16T13:16:23.176689+0000', 'modified': '2024-09-16T13:17:07.012782+0000', 'last_up_change': '2024-09-16T13:16:33.765413+0000', 'last_in_change': '2024-09-16T13:16:25.201328+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:16:33.497561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:16:40.994538+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-16T13:16:50.759705+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 34, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'be8543b4-3d28-4b37-a4ce-d2a56a09aeaf', 'up': 1, 'in': 1, 'weight': 0.8320770263671875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6808', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6809', 'nonce': 22271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6810', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6811', 'nonce': 22271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6816', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6817', 'nonce': 22271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6813', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6814', 'nonce': 22271}]}, 'public_addr': '172.21.15.98:6809/22271', 'cluster_addr': '172.21.15.98:6811/22271', 'heartbeat_back_addr': '172.21.15.98:6817/22271', 'heartbeat_front_addr': '172.21.15.98:6814/22271', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a7f95c7b-56d4-437b-a160-cabb8fd2b1e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6812', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6815', 'nonce': 22272}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6818', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6819', 'nonce': 22272}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6822', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6823', 'nonce': 22272}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6820', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6821', 'nonce': 22272}]}, 'public_addr': '172.21.15.98:6815/22272', 'cluster_addr': '172.21.15.98:6819/22272', 'heartbeat_back_addr': '172.21.15.98:6823/22272', 'heartbeat_front_addr': '172.21.15.98:6821/22272', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6800', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6801', 'nonce': 22273}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6802', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6803', 'nonce': 22273}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6806', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6807', 'nonce': 22273}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6804', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6805', 'nonce': 22273}]}, 'public_addr': '172.21.15.98:6801/22273', 'cluster_addr': '172.21.15.98:6803/22273', 'heartbeat_back_addr': '172.21.15.98:6807/22273', 'heartbeat_front_addr': '172.21.15.98:6805/22273', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4898b5b5-b912-430c-adcb-c52d90226917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6824', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6825', 'nonce': 22270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6826', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6827', 'nonce': 22270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6830', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6831', 'nonce': 22270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6828', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6829', 'nonce': 22270}]}, 'public_addr': '172.21.15.98:6825/22270', 'cluster_addr': '172.21.15.98:6827/22270', 'heartbeat_back_addr': '172.21.15.98:6831/22270', 'heartbeat_front_addr': '172.21.15.98:6829/22270', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6c478be2-f606-41d8-967c-9a689f09a440', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 22286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 22286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 22286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 22286}]}, 'public_addr': '172.21.15.151:6825/22286', 'cluster_addr': '172.21.15.151:6827/22286', 'heartbeat_back_addr': '172.21.15.151:6831/22286', 'heartbeat_front_addr': '172.21.15.151:6829/22286', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5001c828-b223-4d2c-b893-f9c56da73b83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 30, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 22285}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 22285}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 22285}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 22285}]}, 'public_addr': '172.21.15.151:6809/22285', 'cluster_addr': '172.21.15.151:6811/22285', 'heartbeat_back_addr': '172.21.15.151:6815/22285', 'heartbeat_front_addr': '172.21.15.151:6813/22285', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6f7a9125-674c-4be2-93e1-b6296cd00f8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 22315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 22315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 22315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 22315}]}, 'public_addr': '172.21.15.151:6817/22315', 'cluster_addr': '172.21.15.151:6819/22315', 'heartbeat_back_addr': '172.21.15.151:6823/22315', 'heartbeat_front_addr': '172.21.15.151:6821/22315', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bcdfe435-f24d-45a9-a629-7380ad2e575a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6800', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6801', 'nonce': 22284}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 22284}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 22284}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 22284}]}, 'public_addr': '172.21.15.151:6801/22284', 'cluster_addr': '172.21.15.151:6803/22284', 'heartbeat_back_addr': '172.21.15.151:6807/22284', 'heartbeat_front_addr': '172.21.15.151:6805/22284', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.191311+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.563551+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.493457+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.599815+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.634352+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.262873+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.508881+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.348276+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T13:17:07.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:17:07.452 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:07.452 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:07.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:07.452 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:07.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.452 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:830 3.9 e27/23 3:940847c8:::rbd_data.112de98401a1.000000000000001b:head v 27'6, mlcod=27'4)", 2024-09-16T13:17:07.452 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.244050+0000", 2024-09-16T13:17:07.452 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.1952166979999994, 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017733099999999999, 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244050+0000", 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244050+0000", 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244052+0000", 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9879999999999999e-06 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244158+0000", 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105241 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244190+0000", 2024-09-16T13:17:07.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2484999999999998e-05 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244202+0000", 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1722999999999999e-05 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244295+0000", 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3566999999999994e-05 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.244326+0000", 2024-09-16T13:17:07.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0849000000000002e-05 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.245722+0000", 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001396331 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.245824+0000", 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010112599999999999 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:906 3.0 e27/23 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head v 27'7, mlcod=27'5)", 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.428930+0000", 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0103365879999995, 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020851440000000001, 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428930+0000", 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428930+0000", 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.428932+0000", 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9570000000000001e-06 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.429019+0000", 2024-09-16T13:17:07.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6584999999999999e-05 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.429055+0000", 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5573e-05 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.429068+0000", 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3362e-05 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.429196+0000", 2024-09-16T13:17:07.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000128558 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.429260+0000", 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3328000000000001e-05 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.430924+0000", 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001664366 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.431016+0000", 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1415000000000005e-05 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:989 3.0 e27/23 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head v 27'8, mlcod=27'6)", 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.647434+0000", 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 8.7918328020000001, 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021370790000000001, 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.461 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647434+0000", 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647434+0000", 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647436+0000", 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7799999999999999e-06 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647502+0000", 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6135000000000003e-05 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647535+0000", 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3315999999999997e-05 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647547+0000", 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1866e-05 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647685+0000", 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013769699999999999 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.647739+0000", 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3699000000000002e-05 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.649487+0000", 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017479240000000001 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.649571+0000", 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4661999999999997e-05 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:990 3.0 e27/23 3:075c32df:::rbd_data.112de98401a1.000000000000000c:head v 27'9, mlcod=27'7)", 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:58.651339+0000", 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 8.7879278159999998, 2024-09-16T13:17:07.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018613659999999999, 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651339+0000", 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651339+0000", 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651340+0000", 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.297e-06 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651367+0000", 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6376e-05 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651392+0000", 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.472e-05 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651402+0000", 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0232e-05 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651513+0000", 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000111535 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.651584+0000", 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0204999999999996e-05 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.653130+0000", 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015469749999999999 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:58.653201+0000", 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0025999999999997e-05 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1264 3.4 e27/23 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head v 27'4, mlcod=26'2)", 2024-09-16T13:17:07.469 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.9380182020000003, 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026410209999999999, 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501250+0000", 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2410000000000001e-06 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501259+0000", 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501277+0000", 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8683999999999999e-05 2024-09-16T13:17:07.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501288+0000", 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.065e-05 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501562+0000", 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027421200000000002 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501618+0000", 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6045000000000003e-05 2024-09-16T13:17:07.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503841+0000", 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022230829999999998 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503890+0000", 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8526000000000003e-05 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1263 3.0 e27/23 3:0214ce37:::rbd_data.112de98401a1.0000000000000007:head v 27'12, mlcod=27'10)", 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.501307+0000", 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.9379596159999997, 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018975210000000001, 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501307+0000", 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501307+0000", 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501308+0000", 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1199999999999998e-07 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501316+0000", 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1599999999999998e-06 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501323+0000", 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.233e-06 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501328+0000", 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7099999999999998e-06 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501618+0000", 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028967799999999997 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501671+0000", 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3869999999999998e-05 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503092+0000", 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014201859999999999 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503205+0000", 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113272 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1906 3.9 e31/23 3:90f1252a:::rbd_data.112d2ce070e7.000000000000002e:head v 31'10, mlcod=27'7)", 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.015221+0000", 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4240460759999998, 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001825651, 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015221+0000", 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015221+0000", 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015221+0000", 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.28e-07 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015225+0000", 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8199999999999998e-06 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015233+0000", 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6690000000000006e-06 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015239+0000", 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2580000000000001e-06 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015316+0000", 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7392999999999996e-05 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.015341+0000", 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4258000000000001e-05 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.017010+0000", 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016689999999999999 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.017047+0000", 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7024999999999999e-05 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2361 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'23, mlcod=31'21)", 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.864739+0000", 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.574527631, 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019981550000000002, 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864739+0000", 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864739+0000", 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864741+0000", 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.866e-06 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864790+0000", 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9199000000000001e-05 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864836+0000", 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5108999999999998e-05 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864848+0000", 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2193e-05 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.864972+0000", 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000124024 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.865047+0000", 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5074999999999995e-05 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.866622+0000", 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001575629 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.866737+0000", 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011506 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2391 3.0 e31/23 3:01a02ab3:::rbd_data.112dbfd5d823.000000000000000c:head v 31'24, mlcod=31'22)", 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:03.898493+0000", 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.5407737409999998, 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001790378, 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898493+0000", 2024-09-16T13:17:07.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898493+0000", 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898495+0000", 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.165e-06 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898522+0000", 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7010999999999999e-05 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898558+0000", 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5476999999999999e-05 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898571+0000", 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3478999999999999e-05 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898680+0000", 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109064 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.898732+0000", 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1385999999999999e-05 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.900202+0000", 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001469852 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:03.900284+0000", 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1944e-05 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.431874514, 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.431089015, 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:17:07.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.4299220930000001, 2024-09-16T13:17:07.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:17:07.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.499 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:07.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:17:07.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:17:07.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.427047462, 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:17:07.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.4240337219999999, 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:17:07.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.398829519, 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:17:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:17:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:17:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:17:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2623 3.1f e31/30 3:ffd0fa40:::rbd_data.112dbfd5d823.0000000000000328:head v 31'14, mlcod=31'12)", 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.184023+0000", 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.2552441000000001, 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001717302, 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184023+0000", 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184023+0000", 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184025+0000", 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184047+0000", 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2313000000000001e-05 2024-09-16T13:17:07.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184088+0000", 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0754999999999999e-05 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184107+0000", 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8957000000000002e-05 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184242+0000", 2024-09-16T13:17:07.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000135248 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.184288+0000", 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5945000000000001e-05 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.185652+0000", 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001363471 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.185740+0000", 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8332999999999996e-05 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2687 3.1f e31/30 3:fac66a9d:::rbd_data.112dbfd5d823.00000000000002fc:head v 31'16, mlcod=31'14)", 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.312031+0000", 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.1272361860000002, 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001749757, 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.312031+0000", 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.312031+0000", 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.312032+0000", 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3400000000000001e-06 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.312156+0000", 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012430899999999999 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.312182+0000", 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5511e-05 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.312197+0000", 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4953999999999999e-05 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.312281+0000", 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4536000000000005e-05 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.312333+0000", 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1155000000000001e-05 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.313701+0000", 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013679339999999999 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.313781+0000", 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0018000000000005e-05 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2753 3.9 e31/23 3:90cee3ad:::rbd_data.112dbfd5d823.00000000000002e9:head v 31'21, mlcod=31'19)", 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.454581+0000", 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.9846864750000002, 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00180893, 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.454581+0000", 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.454581+0000", 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.454582+0000", 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6899999999999999e-06 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.454714+0000", 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013228299999999999 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.454746+0000", 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1260999999999998e-05 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.454756+0000", 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0487e-05 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.454870+0000", 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011367 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.454914+0000", 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4316000000000001e-05 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.456307+0000", 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013929019999999999 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.456389+0000", 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2321e-05 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.530 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2771 3.9 e31/23 3:90cee3ad:::rbd_data.112dbfd5d823.00000000000002e9:head v 31'22, mlcod=31'20)", 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.495286+0000", 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.9439807770000002, 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018618389999999999, 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.495286+0000", 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.495286+0000", 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.495289+0000", 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5900000000000002e-06 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.495308+0000", 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9639999999999998e-05 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.495352+0000", 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3532000000000003e-05 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.495367+0000", 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5325999999999999e-05 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.495487+0000", 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119491 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.495535+0000", 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8001999999999999e-05 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.497061+0000", 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015262159999999999 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.497148+0000", 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7041999999999998e-05 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.534 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.6326806299999999, 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:17:07.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:17:07.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:17:07.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:17:07.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.539 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.539 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.539 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:07.540 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:07.444+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:07.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:07.552 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:07.546+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:07.652 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:07.646+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:07.695 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:07.696 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:07.696 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:07.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:07.753 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:07.748+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:07.763 INFO:teuthology.orchestra.run.smithi098.stdout:869 write 0x382b44a thru 0x382d8eb (0x24a2 bytes) 2024-09-16T13:17:07.776 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:07.780 INFO:teuthology.orchestra.run.smithi098.stdout:872 trunc from 0xdeb7a53 to 0xbdccb65 2024-09-16T13:17:07.810 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:07.810 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:07.810 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:07.810 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:07.810 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:07.813 INFO:teuthology.orchestra.run.smithi098.stdout:873 write 0x1e113c9 thru 0x1e1abf0 (0x9828 bytes) 2024-09-16T13:17:07.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:07.821 INFO:teuthology.orchestra.run.smithi098.stdout:876 write 0x34d45c0 thru 0x34d54d7 (0xf18 bytes) 2024-09-16T13:17:07.838 INFO:teuthology.orchestra.run.smithi098.stdout:877 writesame 0xccf0420 thru 0xccffb37 data_size 0x4(0xf718 bytes) 2024-09-16T13:17:07.838 INFO:teuthology.orchestra.run.smithi098.stdout:882 punch from 0x468c21e to 0x468cc7c, (0xa5e bytes) 2024-09-16T13:17:07.838 INFO:teuthology.orchestra.run.smithi098.stdout:884 write 0x5b64018 thru 0x5b6d4cf (0x94b8 bytes) 2024-09-16T13:17:07.847 INFO:teuthology.orchestra.run.smithi098.stdout:886 writesame 0x13fef5 thru 0x141522 data_size 0x11(0x162e bytes) 2024-09-16T13:17:07.853 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:07.848+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:07.863 INFO:teuthology.orchestra.run.smithi098.stdout:887 trunc from 0xccffb38 to 0x9013374 2024-09-16T13:17:07.914 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:17:07.915 INFO:teuthology.orchestra.run.smithi098.stdout:889 punch from 0x242c7ae to 0x24326d4, (0x5f26 bytes) 2024-09-16T13:17:07.915 INFO:teuthology.orchestra.run.smithi098.stdout:890 read 0x5ec00b8 thru 0x5ec6015 (0x5f5e bytes) 2024-09-16T13:17:07.916 INFO:teuthology.orchestra.run.smithi098.stdout:891 punch from 0x2614128 to 0x26218f5, (0xd7cd bytes) 2024-09-16T13:17:07.916 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:17:07.926 INFO:teuthology.orchestra.run.smithi098.stdout:892 writesame 0x952b478 thru 0x952c0a7 data_size 0x28(0xc30 bytes) 2024-09-16T13:17:07.926 INFO:teuthology.orchestra.run.smithi098.stdout:893 read 0x75dabaf thru 0x75dfc7c (0x50ce bytes) 2024-09-16T13:17:07.926 INFO:teuthology.orchestra.run.smithi098.stdout:894 read 0x37db9f1 thru 0x37eab05 (0xf115 bytes) 2024-09-16T13:17:07.928 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-16T13:17:07.446056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47495457,"num_objects":82,"num_object_clones":7,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":35,"num_read":5697,"num_read_kb":7783,"num_write":1440,"num_write_kb":10466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1143,"ondisk_log_size":1143,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":83,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2411896,"kb_used_data":3908,"kb_used_omap":13,"kb_used_meta":2407858,"kb_avail":752562824,"statfs":{"total":773094113280,"available":770624331776,"internally_reserved":0,"allocated":4001792,"data_stored":7923395,"data_compressed":368235,"data_compressed_allocated":503808,"data_compressed_original":6221698,"omap_allocated":13632,"internal_metadata":2465647296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30253199,"num_objects":65,"num_object_clones":7,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":35,"num_read":2635,"num_read_kb":4961,"num_write":639,"num_write_kb":5009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.510971"},"pg_stats":[{"pgid":"3.1f","version":"31'13","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.154897+0000","last_change":"2024-09-16T13:17:02.968041+0000","last_active":"2024-09-16T13:17:03.154897+0000","last_peered":"2024-09-16T13:17:03.154897+0000","last_clean":"2024-09-16T13:17:03.154897+0000","last_became_active":"2024-09-16T13:17:02.950173+0000","last_became_peered":"2024-09-16T13:17:02.950173+0000","last_unstale":"2024-09-16T13:17:03.154897+0000","last_undegraded":"2024-09-16T13:17:03.154897+0000","last_fullsized":"2024-09-16T13:17:03.154897+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:04:46.146416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054641499999999996,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":2,"num_write":12,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"31'2","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.009129+0000","last_change":"2024-09-16T13:16:54.298169+0000","last_active":"2024-09-16T13:17:03.009129+0000","last_peered":"2024-09-16T13:17:03.009129+0000","last_clean":"2024-09-16T13:17:03.009129+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:17:03.009129+0000","last_undegraded":"2024-09-16T13:17:03.009129+0000","last_fullsized":"2024-09-16T13:17:03.009129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:50.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001135177,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"27'13","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:02.943602+0000","last_change":"2024-09-16T13:16:54.293554+0000","last_active":"2024-09-16T13:17:02.943602+0000","last_peered":"2024-09-16T13:17:02.943602+0000","last_clean":"2024-09-16T13:17:02.943602+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:17:02.943602+0000","last_undegraded":"2024-09-16T13:17:02.943602+0000","last_fullsized":"2024-09-16T13:17:02.943602+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:14.387827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001293439,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"30'11","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.125195+0000","last_change":"2024-09-16T13:16:54.293251+0000","last_active":"2024-09-16T13:17:03.125195+0000","last_peered":"2024-09-16T13:17:03.125195+0000","last_clean":"2024-09-16T13:17:03.125195+0000","last_became_active":"2024-09-16T13:16:52.292334+0000","last_became_peered":"2024-09-16T13:16:52.292334+0000","last_unstale":"2024-09-16T13:17:03.125195+0000","last_undegraded":"2024-09-16T13:17:03.125195+0000","last_fullsized":"2024-09-16T13:17:03.125195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:07.548141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010706929999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":11,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"31'131","reported_seq":161,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:04.461539+0000","last_change":"2024-09-16T13:17:02.955334+0000","last_active":"2024-09-16T13:17:04.461539+0000","last_peered":"2024-09-16T13:17:04.461539+0000","last_clean":"2024-09-16T13:17:04.461539+0000","last_became_active":"2024-09-16T13:17:02.948727+0000","last_became_peered":"2024-09-16T13:17:02.948727+0000","last_unstale":"2024-09-16T13:17:04.461539+0000","last_undegraded":"2024-09-16T13:17:04.461539+0000","last_fullsized":"2024-09-16T13:17:04.461539+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:17:17.238723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011449159999999999,"stat_sum":{"num_bytes":295721,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":660,"num_read_kb":534,"num_write":226,"num_write_kb":266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"31'24","reported_seq":208,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:04.423397+0000","last_change":"2024-09-16T13:16:54.304914+0000","last_active":"2024-09-16T13:17:04.423397+0000","last_peered":"2024-09-16T13:17:04.423397+0000","last_clean":"2024-09-16T13:17:04.423397+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:17:04.423397+0000","last_undegraded":"2024-09-16T13:17:04.423397+0000","last_fullsized":"2024-09-16T13:17:04.423397+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:31.071219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017517260000000001,"stat_sum":{"num_bytes":4161555,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":168,"num_read_kb":311,"num_write":21,"num_write_kb":285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"31'7","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:04.005587+0000","last_change":"2024-09-16T13:16:54.295771+0000","last_active":"2024-09-16T13:17:04.005587+0000","last_peered":"2024-09-16T13:17:04.005587+0000","last_clean":"2024-09-16T13:17:04.005587+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:17:04.005587+0000","last_undegraded":"2024-09-16T13:17:04.005587+0000","last_fullsized":"2024-09-16T13:17:04.005587+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:42:30.327300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012778399999999999,"stat_sum":{"num_bytes":1173586,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":7,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"31'2","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.013999+0000","last_change":"2024-09-16T13:17:02.963249+0000","last_active":"2024-09-16T13:17:03.013999+0000","last_peered":"2024-09-16T13:17:03.013999+0000","last_clean":"2024-09-16T13:17:03.013999+0000","last_became_active":"2024-09-16T13:17:02.962736+0000","last_became_peered":"2024-09-16T13:17:02.962736+0000","last_unstale":"2024-09-16T13:17:03.013999+0000","last_undegraded":"2024-09-16T13:17:03.013999+0000","last_fullsized":"2024-09-16T13:17:03.013999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:27:05.338093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012285180000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:02.998600+0000","last_change":"2024-09-16T13:16:54.304169+0000","last_active":"2024-09-16T13:17:02.998600+0000","last_peered":"2024-09-16T13:17:02.998600+0000","last_clean":"2024-09-16T13:17:02.998600+0000","last_became_active":"2024-09-16T13:16:52.287752+0000","last_became_peered":"2024-09-16T13:16:52.287752+0000","last_unstale":"2024-09-16T13:17:02.998600+0000","last_undegraded":"2024-09-16T13:17:02.998600+0000","last_fullsized":"2024-09-16T13:17:02.998600+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:02:33.400749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087549000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":95,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"27'9","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:02.944059+0000","last_change":"2024-09-16T13:16:54.292979+0000","last_active":"2024-09-16T13:17:02.944059+0000","last_peered":"2024-09-16T13:17:02.944059+0000","last_clean":"2024-09-16T13:17:02.944059+0000","last_became_active":"2024-09-16T13:16:52.296320+0000","last_became_peered":"2024-09-16T13:16:52.296320+0000","last_unstale":"2024-09-16T13:17:02.944059+0000","last_undegraded":"2024-09-16T13:17:02.944059+0000","last_fullsized":"2024-09-16T13:17:02.944059+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:42:35.683007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000822955,"stat_sum":{"num_bytes":3108016,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":553,"num_write":9,"num_write_kb":206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"31'21","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.003074+0000","last_change":"2024-09-16T13:16:54.624096+0000","last_active":"2024-09-16T13:17:03.003074+0000","last_peered":"2024-09-16T13:17:03.003074+0000","last_clean":"2024-09-16T13:17:03.003074+0000","last_became_active":"2024-09-16T13:16:52.293972+0000","last_became_peered":"2024-09-16T13:16:52.293972+0000","last_unstale":"2024-09-16T13:17:03.003074+0000","last_undegraded":"2024-09-16T13:17:03.003074+0000","last_fullsized":"2024-09-16T13:17:03.003074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:36.145926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001372786,"stat_sum":{"num_bytes":3959634,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":20,"num_read_kb":644,"num_write":20,"num_write_kb":395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"31'10","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.571559+0000","last_change":"2024-09-16T13:16:54.305409+0000","last_active":"2024-09-16T13:17:03.571559+0000","last_peered":"2024-09-16T13:17:03.571559+0000","last_clean":"2024-09-16T13:17:03.571559+0000","last_became_active":"2024-09-16T13:16:52.297145+0000","last_became_peered":"2024-09-16T13:16:52.297145+0000","last_unstale":"2024-09-16T13:17:03.571559+0000","last_undegraded":"2024-09-16T13:17:03.571559+0000","last_fullsized":"2024-09-16T13:17:03.571559+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:04.575154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001794908,"stat_sum":{"num_bytes":884736,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:17:01.997450+0000","last_change":"2024-09-16T13:16:45.344339+0000","last_active":"2024-09-16T13:17:01.997450+0000","last_peered":"2024-09-16T13:17:01.997450+0000","last_clean":"2024-09-16T13:17:01.997450+0000","last_became_active":"2024-09-16T13:16:42.825708+0000","last_became_peered":"2024-09-16T13:16:42.825708+0000","last_unstale":"2024-09-16T13:17:01.997450+0000","last_undegraded":"2024-09-16T13:17:01.997450+0000","last_fullsized":"2024-09-16T13:17:01.997450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:23.678862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"31'14","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.041167+0000","last_change":"2024-09-16T13:16:54.303371+0000","last_active":"2024-09-16T13:17:03.041167+0000","last_peered":"2024-09-16T13:17:03.041167+0000","last_clean":"2024-09-16T13:17:03.041167+0000","last_became_active":"2024-09-16T13:16:52.292830+0000","last_became_peered":"2024-09-16T13:16:52.292830+0000","last_unstale":"2024-09-16T13:17:03.041167+0000","last_undegraded":"2024-09-16T13:17:03.041167+0000","last_fullsized":"2024-09-16T13:17:03.041167+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:08:18.175920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043376500000000002,"stat_sum":{"num_bytes":1179694,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":14,"num_read_kb":5,"num_write":12,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:17:01.993930+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:17:01.993930+0000","last_peered":"2024-09-16T13:17:01.993930+0000","last_clean":"2024-09-16T13:17:01.993930+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:17:01.993930+0000","last_undegraded":"2024-09-16T13:17:01.993930+0000","last_fullsized":"2024-09-16T13:17:01.993930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"31'15","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.762697+0000","last_change":"2024-09-16T13:16:54.298553+0000","last_active":"2024-09-16T13:17:03.762697+0000","last_peered":"2024-09-16T13:17:03.762697+0000","last_clean":"2024-09-16T13:17:03.762697+0000","last_became_active":"2024-09-16T13:16:52.294439+0000","last_became_peered":"2024-09-16T13:16:52.294439+0000","last_unstale":"2024-09-16T13:17:03.762697+0000","last_undegraded":"2024-09-16T13:17:03.762697+0000","last_fullsized":"2024-09-16T13:17:03.762697+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:42.121642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013329209999999999,"stat_sum":{"num_bytes":1146880,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:17:01.994537+0000","last_change":"2024-09-16T13:16:45.344887+0000","last_active":"2024-09-16T13:17:01.994537+0000","last_peered":"2024-09-16T13:17:01.994537+0000","last_clean":"2024-09-16T13:17:01.994537+0000","last_became_active":"2024-09-16T13:16:42.821357+0000","last_became_peered":"2024-09-16T13:16:42.821357+0000","last_unstale":"2024-09-16T13:17:01.994537+0000","last_undegraded":"2024-09-16T13:17:01.994537+0000","last_fullsized":"2024-09-16T13:17:01.994537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:11:35.373375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"31'8","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:04.044227+0000","last_change":"2024-09-16T13:16:54.295888+0000","last_active":"2024-09-16T13:17:04.044227+0000","last_peered":"2024-09-16T13:17:04.044227+0000","last_clean":"2024-09-16T13:17:04.044227+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:17:04.044227+0000","last_undegraded":"2024-09-16T13:17:04.044227+0000","last_fullsized":"2024-09-16T13:17:04.044227+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:30.996264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084973399999999999,"stat_sum":{"num_bytes":655360,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":8,"num_write_kb":289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:17:01.993852+0000","last_change":"2024-09-16T13:16:44.845588+0000","last_active":"2024-09-16T13:17:01.993852+0000","last_peered":"2024-09-16T13:17:01.993852+0000","last_clean":"2024-09-16T13:17:01.993852+0000","last_became_active":"2024-09-16T13:16:42.823703+0000","last_became_peered":"2024-09-16T13:16:42.823703+0000","last_unstale":"2024-09-16T13:17:01.993852+0000","last_undegraded":"2024-09-16T13:17:01.993852+0000","last_fullsized":"2024-09-16T13:17:01.993852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:02:15.873787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"31'19","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.001661+0000","last_change":"2024-09-16T13:16:54.293894+0000","last_active":"2024-09-16T13:17:03.001661+0000","last_peered":"2024-09-16T13:17:03.001661+0000","last_clean":"2024-09-16T13:17:03.001661+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:17:03.001661+0000","last_undegraded":"2024-09-16T13:17:03.001661+0000","last_fullsized":"2024-09-16T13:17:03.001661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:34.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015391249999999999,"stat_sum":{"num_bytes":4022341,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":22,"num_read_kb":598,"num_write":18,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"31'14","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.763788+0000","last_change":"2024-09-16T13:16:54.304919+0000","last_active":"2024-09-16T13:17:03.763788+0000","last_peered":"2024-09-16T13:17:03.763788+0000","last_clean":"2024-09-16T13:17:03.763788+0000","last_became_active":"2024-09-16T13:16:52.296291+0000","last_became_peered":"2024-09-16T13:16:52.296291+0000","last_unstale":"2024-09-16T13:17:03.763788+0000","last_undegraded":"2024-09-16T13:17:03.763788+0000","last_fullsized":"2024-09-16T13:17:03.763788+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:32.115766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001425195,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":2,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":27,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:17:01.998036+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:17:01.998036+0000","last_peered":"2024-09-16T13:17:01.998036+0000","last_clean":"2024-09-16T13:17:01.998036+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:17:01.998036+0000","last_undegraded":"2024-09-16T13:17:01.998036+0000","last_fullsized":"2024-09-16T13:17:01.998036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"31'32","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:04.378370+0000","last_change":"2024-09-16T13:16:54.294518+0000","last_active":"2024-09-16T13:17:04.378370+0000","last_peered":"2024-09-16T13:17:04.378370+0000","last_clean":"2024-09-16T13:17:04.378370+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:17:04.378370+0000","last_undegraded":"2024-09-16T13:17:04.378370+0000","last_fullsized":"2024-09-16T13:17:04.378370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:29.090161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046065400000000001,"stat_sum":{"num_bytes":3961067,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":3,"num_read":17,"num_read_kb":35,"num_write":29,"num_write_kb":433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:17:01.997852+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:17:01.997852+0000","last_peered":"2024-09-16T13:17:01.997852+0000","last_clean":"2024-09-16T13:17:01.997852+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:17:01.997852+0000","last_undegraded":"2024-09-16T13:17:01.997852+0000","last_fullsized":"2024-09-16T13:17:01.997852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"27'3","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:02.947020+0000","last_change":"2024-09-16T13:16:54.303675+0000","last_active":"2024-09-16T13:17:02.947020+0000","last_peered":"2024-09-16T13:17:02.947020+0000","last_clean":"2024-09-16T13:17:02.947020+0000","last_became_active":"2024-09-16T13:16:52.296479+0000","last_became_peered":"2024-09-16T13:16:52.296479+0000","last_unstale":"2024-09-16T13:17:02.947020+0000","last_undegraded":"2024-09-16T13:17:02.947020+0000","last_fullsized":"2024-09-16T13:17:02.947020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:27:17.097803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067097399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:17:01.994304+0000","last_change":"2024-09-16T13:16:45.344856+0000","last_active":"2024-09-16T13:17:01.994304+0000","last_peered":"2024-09-16T13:17:01.994304+0000","last_clean":"2024-09-16T13:17:01.994304+0000","last_became_active":"2024-09-16T13:16:42.825917+0000","last_became_peered":"2024-09-16T13:16:42.825917+0000","last_unstale":"2024-09-16T13:17:01.994304+0000","last_undegraded":"2024-09-16T13:17:01.994304+0000","last_fullsized":"2024-09-16T13:17:01.994304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:30.133715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082018499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"31'144","reported_seq":934,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.391204+0000","last_change":"2024-09-16T13:16:54.298588+0000","last_active":"2024-09-16T13:17:03.391204+0000","last_peered":"2024-09-16T13:17:03.391204+0000","last_clean":"2024-09-16T13:17:03.391204+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:17:03.391204+0000","last_undegraded":"2024-09-16T13:17:03.391204+0000","last_fullsized":"2024-09-16T13:17:03.391204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:45.154984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001266124,"stat_sum":{"num_bytes":727067,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2242,"num_read_kb":1969,"num_write":135,"num_write_kb":399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:17:01.997921+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:17:01.997921+0000","last_peered":"2024-09-16T13:17:01.997921+0000","last_clean":"2024-09-16T13:17:01.997921+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:17:01.997921+0000","last_undegraded":"2024-09-16T13:17:01.997921+0000","last_fullsized":"2024-09-16T13:17:01.997921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":113,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T13:17:01.994092+0000","last_change":"2024-09-16T13:16:34.788814+0000","last_active":"2024-09-16T13:17:01.994092+0000","last_peered":"2024-09-16T13:17:01.994092+0000","last_clean":"2024-09-16T13:17:01.994092+0000","last_became_active":"2024-09-16T13:16:34.786680+0000","last_became_peered":"2024-09-16T13:16:34.786680+0000","last_unstale":"2024-09-16T13:17:01.994092+0000","last_undegraded":"2024-09-16T13:17:01.994092+0000","last_fullsized":"2024-09-16T13:17:01.994092+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:27:17.345314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"31'329","reported_seq":353,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:04.185029+0000","last_change":"2024-09-16T13:16:54.297765+0000","last_active":"2024-09-16T13:17:04.185029+0000","last_peered":"2024-09-16T13:17:04.185029+0000","last_clean":"2024-09-16T13:17:04.185029+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:17:04.185029+0000","last_undegraded":"2024-09-16T13:17:04.185029+0000","last_fullsized":"2024-09-16T13:17:04.185029+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:59.515115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012012500000000001,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1512,"num_read_kb":1258,"num_write":523,"num_write_kb":756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"31'21","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:04.019093+0000","last_change":"2024-09-16T13:16:54.303746+0000","last_active":"2024-09-16T13:17:04.019093+0000","last_peered":"2024-09-16T13:17:04.019093+0000","last_clean":"2024-09-16T13:17:04.019093+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:17:04.019093+0000","last_undegraded":"2024-09-16T13:17:04.019093+0000","last_fullsized":"2024-09-16T13:17:04.019093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:49:10.301550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065283599999999995,"stat_sum":{"num_bytes":3736890,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"31'20","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:04.133673+0000","last_change":"2024-09-16T13:16:54.328489+0000","last_active":"2024-09-16T13:17:04.133673+0000","last_peered":"2024-09-16T13:17:04.133673+0000","last_clean":"2024-09-16T13:17:04.133673+0000","last_became_active":"2024-09-16T13:16:52.299458+0000","last_became_peered":"2024-09-16T13:16:52.299458+0000","last_unstale":"2024-09-16T13:17:04.133673+0000","last_undegraded":"2024-09-16T13:17:04.133673+0000","last_fullsized":"2024-09-16T13:17:04.133673+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:34.071306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069184499999999996,"stat_sum":{"num_bytes":1179648,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":4,"num_read_kb":2,"num_write":19,"num_write_kb":168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"31'52","reported_seq":317,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:04.443673+0000","last_change":"2024-09-16T13:16:54.298033+0000","last_active":"2024-09-16T13:17:04.443673+0000","last_peered":"2024-09-16T13:17:04.443673+0000","last_clean":"2024-09-16T13:17:04.443673+0000","last_became_active":"2024-09-16T13:16:52.290772+0000","last_became_peered":"2024-09-16T13:16:52.290772+0000","last_unstale":"2024-09-16T13:17:04.443673+0000","last_undegraded":"2024-09-16T13:17:04.443673+0000","last_fullsized":"2024-09-16T13:17:04.443673+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:39:27.929265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00121396,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":711,"num_read_kb":608,"num_write":51,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"28'13","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:02.943349+0000","last_change":"2024-09-16T13:16:54.292717+0000","last_active":"2024-09-16T13:17:02.943349+0000","last_peered":"2024-09-16T13:17:02.943349+0000","last_clean":"2024-09-16T13:17:02.943349+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:17:02.943349+0000","last_undegraded":"2024-09-16T13:17:02.943349+0000","last_fullsized":"2024-09-16T13:17:02.943349+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:39.937083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065643500000000003,"stat_sum":{"num_bytes":3779714,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":606,"num_write":13,"num_write_kb":292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"31'30","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.023299+0000","last_change":"2024-09-16T13:16:54.624013+0000","last_active":"2024-09-16T13:17:03.023299+0000","last_peered":"2024-09-16T13:17:03.023299+0000","last_clean":"2024-09-16T13:17:03.023299+0000","last_became_active":"2024-09-16T13:16:52.294924+0000","last_became_peered":"2024-09-16T13:16:52.294924+0000","last_unstale":"2024-09-16T13:17:03.023299+0000","last_undegraded":"2024-09-16T13:17:03.023299+0000","last_fullsized":"2024-09-16T13:17:03.023299+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:49:58.037805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012996780000000001,"stat_sum":{"num_bytes":1518319,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":23,"num_read_kb":319,"num_write":28,"num_write_kb":552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"31'18","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.048926+0000","last_change":"2024-09-16T13:16:54.623553+0000","last_active":"2024-09-16T13:17:03.048926+0000","last_peered":"2024-09-16T13:17:03.048926+0000","last_clean":"2024-09-16T13:17:03.048926+0000","last_became_active":"2024-09-16T13:16:52.299462+0000","last_became_peered":"2024-09-16T13:16:52.299462+0000","last_unstale":"2024-09-16T13:17:03.048926+0000","last_undegraded":"2024-09-16T13:17:03.048926+0000","last_fullsized":"2024-09-16T13:17:03.048926+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:23:35.582311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087311099999999998,"stat_sum":{"num_bytes":1933359,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":9,"num_read_kb":3,"num_write":17,"num_write_kb":311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"31'33","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.054172+0000","last_change":"2024-09-16T13:16:54.624805+0000","last_active":"2024-09-16T13:17:03.054172+0000","last_peered":"2024-09-16T13:17:03.054172+0000","last_clean":"2024-09-16T13:17:03.054172+0000","last_became_active":"2024-09-16T13:16:52.299393+0000","last_became_peered":"2024-09-16T13:16:52.299393+0000","last_unstale":"2024-09-16T13:17:03.054172+0000","last_undegraded":"2024-09-16T13:17:03.054172+0000","last_fullsized":"2024-09-16T13:17:03.054172+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:20.663757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019518529999999999,"stat_sum":{"num_bytes":4161574,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":4,"num_read":123,"num_read_kb":94,"num_write":52,"num_write_kb":2323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"31'16","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:04.006927+0000","last_change":"2024-09-16T13:16:54.328263+0000","last_active":"2024-09-16T13:17:04.006927+0000","last_peered":"2024-09-16T13:17:04.006927+0000","last_clean":"2024-09-16T13:17:04.006927+0000","last_became_active":"2024-09-16T13:16:52.294061+0000","last_became_peered":"2024-09-16T13:16:52.294061+0000","last_unstale":"2024-09-16T13:17:04.006927+0000","last_undegraded":"2024-09-16T13:17:04.006927+0000","last_fullsized":"2024-09-16T13:17:04.006927+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:43:32.004580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049957799999999998,"stat_sum":{"num_bytes":4140199,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":11,"num_read_kb":173,"num_write":14,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"31'11","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.003030+0000","last_change":"2024-09-16T13:16:54.625008+0000","last_active":"2024-09-16T13:17:03.003030+0000","last_peered":"2024-09-16T13:17:03.003030+0000","last_clean":"2024-09-16T13:17:03.003030+0000","last_became_active":"2024-09-16T13:16:52.300728+0000","last_became_peered":"2024-09-16T13:16:52.300728+0000","last_unstale":"2024-09-16T13:17:03.003030+0000","last_undegraded":"2024-09-16T13:17:03.003030+0000","last_fullsized":"2024-09-16T13:17:03.003030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:53.877292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001790673,"stat_sum":{"num_bytes":1179648,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:02.947067+0000","last_change":"2024-09-16T13:16:54.303399+0000","last_active":"2024-09-16T13:17:02.947067+0000","last_peered":"2024-09-16T13:17:02.947067+0000","last_clean":"2024-09-16T13:17:02.947067+0000","last_became_active":"2024-09-16T13:16:52.293657+0000","last_became_peered":"2024-09-16T13:16:52.293657+0000","last_unstale":"2024-09-16T13:17:02.947067+0000","last_undegraded":"2024-09-16T13:17:02.947067+0000","last_fullsized":"2024-09-16T13:17:02.947067+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:22.641403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049211100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"31'27","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T13:17:03.020942+0000","last_change":"2024-09-16T13:16:54.623401+0000","last_active":"2024-09-16T13:17:03.020942+0000","last_peered":"2024-09-16T13:17:03.020942+0000","last_clean":"2024-09-16T13:17:03.020942+0000","last_became_active":"2024-09-16T13:16:52.299393+0000","last_became_peered":"2024-09-16T13:16:52.299393+0000","last_unstale":"2024-09-16T13:17:03.020942+0000","last_undegraded":"2024-09-16T13:17:03.020942+0000","last_fullsized":"2024-09-16T13:17:03.020942+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:08:01.819612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079151600000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":46905070,"num_objects":78,"num_object_clones":7,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":35,"num_read":5631,"num_read_kb":7728,"num_write":1337,"num_write_kb":8836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":6310388,"data_compressed":341817,"data_compressed_allocated":471040,"data_compressed_original":5582722,"omap_allocated":0,"internal_metadata":0},"log_size":1068,"ondisk_log_size":1068,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65840,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509450,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301852,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069988,"statfs":{"total":96636764160,"available":96327667712,"internally_reserved":0,"allocated":344064,"data_stored":141671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]}]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302064,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069776,"statfs":{"total":96636764160,"available":96327450624,"internally_reserved":0,"allocated":557056,"data_stored":2516745,"data_compressed":135708,"data_compressed_allocated":176128,"data_compressed_original":2359296,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]}]},{"osd":3,"up_from":15,"seq":64424509450,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302060,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069780,"statfs":{"total":96636764160,"available":96327454720,"internally_reserved":0,"allocated":548864,"data_stored":351786,"data_compressed":4509,"data_compressed_allocated":12288,"data_compressed_original":55233,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.083}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1579999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301984,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069856,"statfs":{"total":96636764160,"available":96327532544,"internally_reserved":0,"allocated":483328,"data_stored":331169,"data_compressed":15570,"data_compressed_allocated":24576,"data_compressed_original":110592,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]},{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302256,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069584,"statfs":{"total":96636764160,"available":96327254016,"internally_reserved":0,"allocated":753664,"data_stored":3314381,"data_compressed":174845,"data_compressed_allocated":233472,"data_compressed_original":3024833,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3239999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3140000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4990000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302052,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069788,"statfs":{"total":96636764160,"available":96327462912,"internally_reserved":0,"allocated":544768,"data_stored":369995,"data_compressed":3322,"data_compressed_allocated":8192,"data_compressed_original":69632,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]}]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297704,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074136,"statfs":{"total":96636764160,"available":96331915264,"internally_reserved":0,"allocated":356352,"data_stored":165563,"data_compressed":1361,"data_compressed_allocated":4096,"data_compressed_original":28672,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542153,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301924,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069916,"statfs":{"total":96636764160,"available":96327593984,"internally_reserved":0,"allocated":413696,"data_stored":732085,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":11477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":255115,"data_compressed":19359,"data_compressed_allocated":28672,"data_compressed_original":233472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":35323,"data_compressed":1361,"data_compressed_allocated":4096,"data_compressed_original":28672,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":327192,"data_compressed":6446,"data_compressed_allocated":16384,"data_compressed_original":96193,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":2830280,"data_compressed":169404,"data_compressed_allocated":225280,"data_compressed_original":2703360,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":239736,"data_compressed":3322,"data_compressed_allocated":8192,"data_compressed_original":69632,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":11431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":2599834,"data_compressed":141925,"data_compressed_allocated":188416,"data_compressed_original":2451393,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:17:07.931 INFO:tasks.thrashosds.thrasher:Setting 3.1c to [6, 1] 2024-09-16T13:17:07.931 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.1c', '6', '1'] 2024-09-16T13:17:07.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.1c 6 1 2024-09-16T13:17:07.934 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:07.934 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5098 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777238+0000", 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.15008299899999999, 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012447827999999999, 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5098 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.935 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777238+0000", 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777238+0000", 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777239+0000", 2024-09-16T13:17:07.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8700000000000005e-07 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777244+0000", 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4310000000000002e-06 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777255+0000", 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0628000000000001e-05 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777261+0000", 2024-09-16T13:17:07.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.489e-06 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.781626+0000", 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043644759999999999 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.781730+0000", 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104038 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.782148+0000", 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041808399999999999 2024-09-16T13:17:07.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787937+0000", 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057889860000000003 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789604+0000", 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001666931 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789650+0000", 2024-09-16T13:17:07.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6201e-05 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789686+0000", 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5777e-05 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5099 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777268+0000", 2024-09-16T13:17:07.940 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.15005251999999999, 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013450329, 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5099 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777268+0000", 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777268+0000", 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777269+0000", 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4300000000000003e-07 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777273+0000", 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2420000000000002e-06 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777282+0000", 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3839999999999997e-06 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777287+0000", 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1529999999999996e-06 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.782725+0000", 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005438404 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.782814+0000", 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8968000000000006e-05 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.783212+0000", 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039762899999999998 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787967+0000", 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047552779999999999 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790635+0000", 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002667633 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790682+0000", 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7506000000000002e-05 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790719+0000", 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6489000000000002e-05 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.945 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5100 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777297+0000", 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.15002374900000001, 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013509412, 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5100 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.946 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777297+0000", 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777297+0000", 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777298+0000", 2024-09-16T13:17:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8199999999999999e-07 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777302+0000", 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1590000000000003e-06 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777311+0000", 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9760000000000001e-06 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777319+0000", 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4789999999999996e-06 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.783694+0000", 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063747259999999998 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.783788+0000", 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3647000000000006e-05 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.784272+0000", 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048395399999999998 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788002+0000", 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003729937 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790730+0000", 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027281829999999999 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790772+0000", 2024-09-16T13:17:07.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:17:07.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790807+0000", 2024-09-16T13:17:07.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4096e-05 2024-09-16T13:17:07.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5101 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.14999227600000001, 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595248, 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5101 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.952 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:07.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777333+0000", 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1949999999999996e-06 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777342+0000", 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6950000000000006e-06 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777348+0000", 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785241+0000", 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078929329999999995 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785391+0000", 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150113 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785969+0000", 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057822899999999998 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789745+0000", 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037756529999999999 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793823+0000", 2024-09-16T13:17:07.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040786299999999998 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793880+0000", 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7157999999999997e-05 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793924+0000", 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3407e-05 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5102 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.149944984, 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016663710000000002, 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5102 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:07.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:07.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777377+0000", 2024-09-16T13:17:07.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4600000000000003e-07 2024-09-16T13:17:07.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777381+0000", 2024-09-16T13:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5290000000000002e-06 2024-09-16T13:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777393+0000", 2024-09-16T13:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2001000000000001e-05 2024-09-16T13:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777401+0000", 2024-09-16T13:17:07.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3869999999999997e-06 2024-09-16T13:17:07.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786753+0000", 2024-09-16T13:17:07.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093523649999999996 2024-09-16T13:17:07.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786860+0000", 2024-09-16T13:17:07.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106921 2024-09-16T13:17:07.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787286+0000", 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042651800000000003 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793815+0000", 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065290579999999999 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793932+0000", 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116688 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794002+0000", 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0313000000000003e-05 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794040+0000", 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7083999999999999e-05 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5103 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.14988409799999999, 2024-09-16T13:17:07.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017505544000000001, 2024-09-16T13:17:07.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.968 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.968 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.968 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.968 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5103 2024-09-16T13:17:07.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.968 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777438+0000", 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.78e-07 2024-09-16T13:17:07.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777442+0000", 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5950000000000001e-06 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777453+0000", 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0372e-05 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777458+0000", 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6559999999999997e-06 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788034+0000", 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010575753 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788111+0000", 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7038000000000003e-05 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788463+0000", 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000352028 2024-09-16T13:17:07.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794263+0000", 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005800342 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794837+0000", 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057334300000000003 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794897+0000", 2024-09-16T13:17:07.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0690000000000003e-05 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794942+0000", 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4848999999999997e-05 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5104 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:07.973 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.14983969, 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017694142999999999, 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5104 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:07.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777482+0000", 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777487+0000", 2024-09-16T13:17:07.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6639999999999997e-06 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777496+0000", 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2890000000000003e-06 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777501+0000", 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2730000000000002e-06 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788951+0000", 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011449429000000001 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789007+0000", 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5643999999999998e-05 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789205+0000", 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019820800000000001 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:07.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794268+0000", 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050636850000000001 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795094+0000", 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082569700000000002 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795139+0000", 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5055000000000001e-05 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795175+0000", 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6149000000000002e-05 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5105 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.149777349, 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017729482000000001, 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.979 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5105 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777544+0000", 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1100000000000004e-07 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777549+0000", 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777561+0000", 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1524e-05 2024-09-16T13:17:07.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777566+0000", 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.558e-06 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789781+0000", 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012214981 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789870+0000", 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9171000000000004e-05 2024-09-16T13:17:07.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790313+0000", 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044255299999999999 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794273+0000", 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039596129999999998 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795213+0000", 2024-09-16T13:17:07.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094086099999999998 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795239+0000", 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5324e-05 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795273+0000", 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4198e-05 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5106 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.14973220500000001, 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018621867, 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5106 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777590+0000", 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777594+0000", 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6750000000000001e-06 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777604+0000", 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0076e-05 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777610+0000", 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7529999999999998e-06 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790893+0000", 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013282845 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790999+0000", 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106289 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791493+0000", 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049357800000000005 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794662+0000", 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031691409999999999 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796146+0000", 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00148375 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796176+0000", 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0349e-05 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796211+0000", 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4530000000000003e-05 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5107 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.149700944, 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018745067000000001, 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.990 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5107 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:07.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777621+0000", 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-07 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777624+0000", 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7950000000000001e-06 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777633+0000", 2024-09-16T13:17:07.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5979999999999997e-06 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777638+0000", 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.989e-06 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792024+0000", 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014385692 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792104+0000", 2024-09-16T13:17:07.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0567000000000003e-05 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792340+0000", 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023624200000000001 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794667+0000", 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023269990000000002 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:07.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796304+0000", 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001636981 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796331+0000", 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6492999999999999e-05 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796365+0000", 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4102e-05 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:07.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5108 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.149647115, 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019090617000000001, 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5108 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:07.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:17:07.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777677+0000", 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7879999999999998e-06 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777686+0000", 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7679999999999997e-06 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777689+0000", 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.123e-06 2024-09-16T13:17:07.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792751+0000", 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01506158 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792855+0000", 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010456 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:07.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793305+0000", 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045006100000000003 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794671+0000", 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001366304 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796393+0000", 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001721272 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:08.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796733+0000", 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000340611 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796764+0000", 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.099e-05 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5207 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879294+0000", 2024-09-16T13:17:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.048026863000000003, 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012675076, 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5207 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879294+0000", 2024-09-16T13:17:08.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879294+0000", 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879295+0000", 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1960000000000001e-06 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:08.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879302+0000", 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2280000000000001e-06 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879318+0000", 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5224e-05 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879326+0000", 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.887394+0000", 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0080676059999999997 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.887481+0000", 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7264999999999999e-05 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.887889+0000", 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00040844500000000001 2024-09-16T13:17:08.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890217+0000", 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023273170000000002 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891919+0000", 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001702161 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:08.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891947+0000", 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8608000000000001e-05 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891969+0000", 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1679999999999999e-05 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5208 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.047887664000000003, 2024-09-16T13:17:08.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014882177999999999, 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5208 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879434+0000", 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.322e-06 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879443+0000", 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879459+0000", 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6133999999999998e-05 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879467+0000", 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2400000000000007e-06 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888339+0000", 2024-09-16T13:17:08.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088721619999999994 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888461+0000", 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121341 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888819+0000", 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035782799999999998 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891271+0000", 2024-09-16T13:17:08.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002452872 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894224+0000", 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029522910000000001 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894275+0000", 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1647999999999997e-05 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:08.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894315+0000", 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9993999999999997e-05 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5209 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879927+0000", 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.047393561000000001, 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014547783, 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:08.013 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5209 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879927+0000", 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879927+0000", 2024-09-16T13:17:08.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879928+0000", 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2020000000000001e-06 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879936+0000", 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6520000000000006e-06 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879953+0000", 2024-09-16T13:17:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7099999999999999e-05 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879962+0000", 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4840000000000004e-06 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.889353+0000", 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093915830000000002 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:08.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.889418+0000", 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5188e-05 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.889634+0000", 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021556599999999999 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894336+0000", 2024-09-16T13:17:08.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047024179999999999 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894397+0000", 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0817999999999999e-05 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894440+0000", 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3120000000000001e-05 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894475+0000", 2024-09-16T13:17:08.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4652e-05 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5210 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.047332475999999998, 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01466423, 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:08.019 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5210 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879989+0000", 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8800000000000001e-07 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879995+0000", 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880017+0000", 2024-09-16T13:17:08.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2101000000000001e-05 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880024+0000", 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9240000000000001e-06 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890250+0000", 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010226012 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890344+0000", 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.446e-05 2024-09-16T13:17:08.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890770+0000", 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042575099999999998 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894344+0000", 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003574163 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894549+0000", 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020477599999999999 2024-09-16T13:17:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894606+0000", 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6891000000000001e-05 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894653+0000", 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6867999999999999e-05 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5211 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.047288413000000001, 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014800604, 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5211 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880033+0000", 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.06e-07 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880039+0000", 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880052+0000", 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-05 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880059+0000", 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3090000000000001e-06 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891316+0000", 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011257118999999999 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891374+0000", 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8020999999999999e-05 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891577+0000", 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020371699999999999 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894348+0000", 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027708300000000002 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894726+0000", 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037819399999999999 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894784+0000", 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7667999999999997e-05 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894833+0000", 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8936e-05 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5212 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.046933039000000003, 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014645704000000001, 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5212 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:08.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880389+0000", 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1680000000000001e-06 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880395+0000", 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880408+0000", 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.308e-05 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880414+0000", 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.105e-06 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892037+0000", 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011622402 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892116+0000", 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9939999999999997e-05 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892597+0000", 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048076500000000002 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894492+0000", 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001895246 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894921+0000", 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000428114 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894985+0000", 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4777999999999995e-05 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:08.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895034+0000", 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8133999999999997e-05 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5213 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.046799888999999997, 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014632608, 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:08.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5213 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880522+0000", 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2600000000000001e-07 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880527+0000", 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880538+0000", 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917999999999999e-05 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880543+0000", 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.361e-06 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893069+0000", 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012525497999999999 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893152+0000", 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2805000000000004e-05 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:08.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893635+0000", 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048315100000000002 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894852+0000", 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001216806 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895075+0000", 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022300299999999999 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895122+0000", 2024-09-16T13:17:08.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7077999999999999e-05 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895154+0000", 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1919999999999999e-05 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5214 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.046645237999999999, 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017785977000000001, 2024-09-16T13:17:08.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5214 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:08.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880681+0000", 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:08.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880689+0000", 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3610000000000001e-06 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880694+0000", 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5809999999999998e-06 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895186+0000", 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014492430000000001 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895311+0000", 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012518300000000001 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895760+0000", 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044864900000000002 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898337+0000", 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025765229999999998 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898354+0000", 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7538999999999999e-05 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898406+0000", 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2054e-05 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898462+0000", 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5404e-05 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5215 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.046380404, 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018191319000000001, 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5215 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:08.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880941+0000", 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-16T13:17:08.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880946+0000", 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3710000000000002e-06 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880955+0000", 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1230000000000006e-06 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880959+0000", 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896782+0000", 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015822716000000001 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896876+0000", 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3667999999999998e-05 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.897289+0000", 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041357399999999999 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898474+0000", 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011847209999999999 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899036+0000", 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056215699999999998 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899090+0000", 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3662999999999998e-05 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:08.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:08.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899132+0000", 2024-09-16T13:17:08.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1814000000000003e-05 2024-09-16T13:17:08.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.051 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:08.051 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:08.051 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:08.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:17:08.053 INFO:teuthology.orchestra.run.smithi098.stdout:895 trunc from 0x952c0a8 to 0x2ead01b 2024-09-16T13:17:08.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:08.054 INFO:teuthology.orchestra.run.smithi098.stdout:897 write 0x7af85ee thru 0x7afe462 (0x5e75 bytes) 2024-09-16T13:17:08.054 INFO:teuthology.orchestra.run.smithi098.stdout:898 read 0x44176ce thru 0x441a3d9 (0x2d0c bytes) 2024-09-16T13:17:08.054 INFO:teuthology.orchestra.run.smithi098.stdout:899 read 0x11758cf thru 0x117cd57 (0x7489 bytes) 2024-09-16T13:17:08.055 INFO:teuthology.orchestra.run.smithi098.stdout:900 trunc from 0x7afe463 to 0x389717 2024-09-16T13:17:08.055 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:08.044+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:08.077 INFO:teuthology.orchestra.run.smithi098.stdout:902 write 0x6d6c11a thru 0x6d70bc7 (0x4aae bytes) 2024-09-16T13:17:08.085 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:08.085 INFO:teuthology.orchestra.run.smithi098.stdout:907 read 0x2e3d546 thru 0x2e47a8c (0xa547 bytes) 2024-09-16T13:17:08.085 INFO:teuthology.orchestra.run.smithi098.stdout:908 punch from 0x1ef0295 to 0x1ef92b1, (0x901c bytes) 2024-09-16T13:17:08.085 INFO:teuthology.orchestra.run.smithi098.stdout:909 read 0x1145689 thru 0x1154d79 (0xf6f1 bytes) 2024-09-16T13:17:08.102 INFO:teuthology.orchestra.run.smithi098.stdout:910 writesame 0x9f516da thru 0x9f5cf4e data_size 0x1(0xb875 bytes) 2024-09-16T13:17:08.102 INFO:teuthology.orchestra.run.smithi098.stdout:911 write 0x3c344ff thru 0x3c371ab (0x2cad bytes) 2024-09-16T13:17:08.107 INFO:teuthology.orchestra.run.smithi098.stdout:912 read 0x5e7f5e3 thru 0x5e8c992 (0xd3b0 bytes) 2024-09-16T13:17:08.125 INFO:teuthology.orchestra.run.smithi098.stdout:913 write 0xe7c2a49 thru 0xe7c6db8 (0x4370 bytes) 2024-09-16T13:17:08.132 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:08.132 INFO:teuthology.orchestra.run.smithi098.stdout:915 writesame 0xadd4cba thru 0xadde568 data_size 0x2b(0x98af bytes) 2024-09-16T13:17:08.142 INFO:teuthology.orchestra.run.smithi098.stdout:917 write 0x109ffd1 thru 0x10a5bbd (0x5bed bytes) 2024-09-16T13:17:08.150 INFO:teuthology.orchestra.run.smithi098.stdout:918 writesame 0xb7c9c93 thru 0xb7ca702 data_size 0x8(0xa70 bytes) 2024-09-16T13:17:08.153 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:08.146+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:08.155 INFO:teuthology.orchestra.run.smithi098.stdout:919 punch from 0x7cd4a15 to 0x7cd7218, (0x2803 bytes) 2024-09-16T13:17:08.155 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:08.155 INFO:teuthology.orchestra.run.smithi098.stdout:922 write 0xddef4d thru 0xded580 (0xe634 bytes) 2024-09-16T13:17:08.173 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:08.173 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:08.177 INFO:teuthology.orchestra.run.smithi098.stdout:924 trunc from 0xe7c6db9 to 0x6f69398 2024-09-16T13:17:08.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:08.198 INFO:teuthology.orchestra.run.smithi098.stdout:925 trunc from 0x6f69398 to 0x4c559c4 2024-09-16T13:17:08.216 INFO:teuthology.orchestra.run.smithi098.stdout:926 read 0x136cce4 thru 0x137603f (0x935c bytes) 2024-09-16T13:17:08.216 INFO:teuthology.orchestra.run.smithi098.stdout:927 punch from 0x28c4e7 to 0x29aada, (0xe5f3 bytes) 2024-09-16T13:17:08.217 INFO:teuthology.orchestra.run.smithi098.stdout:929 punch from 0x335fd3e to 0x336acc6, (0xaf88 bytes) 2024-09-16T13:17:08.217 INFO:teuthology.orchestra.run.smithi098.stdout:930 punch from 0xae1946 to 0xae5133, (0x37ed bytes) 2024-09-16T13:17:08.217 INFO:teuthology.orchestra.run.smithi098.stdout:931 read 0x133f40a thru 0x1341ebc (0x2ab3 bytes) 2024-09-16T13:17:08.217 INFO:teuthology.orchestra.run.smithi098.stdout:932 punch from 0x3368031 to 0x3377a2b, (0xf9fa bytes) 2024-09-16T13:17:08.217 INFO:teuthology.orchestra.run.smithi098.stdout:933 read 0x1dd0ae5 thru 0x1dd2a3f (0x1f5b bytes) 2024-09-16T13:17:08.217 INFO:teuthology.orchestra.run.smithi098.stdout:934 punch from 0x109090e to 0x1095738, (0x4e2a bytes) 2024-09-16T13:17:08.219 INFO:teuthology.orchestra.run.smithi098.stdout:935 trunc from 0x4c559c4 to 0x317de65 2024-09-16T13:17:08.238 INFO:teuthology.orchestra.run.smithi098.stdout:936 read 0x1d31a52 thru 0x1d3642e (0x49dd bytes) 2024-09-16T13:17:08.238 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:08.242 INFO:teuthology.orchestra.run.smithi098.stdout:939 trunc from 0x317de65 to 0xc307b5 2024-09-16T13:17:08.254 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:08.248+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:08.263 INFO:teuthology.orchestra.run.smithi098.stdout:941 read 0x7f51ef thru 0x7fd68d (0x849f bytes) 2024-09-16T13:17:08.277 INFO:teuthology.orchestra.run.smithi098.stdout:942 trunc from 0xc307b5 to 0x8c89d7d 2024-09-16T13:17:08.284 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:08.285 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:08.285 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:08.285 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:08.285 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:08.288 INFO:teuthology.orchestra.run.smithi098.stdout:943 read 0x675b0fb thru 0x675bdd5 (0xcdb bytes) 2024-09-16T13:17:08.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:08.309 INFO:teuthology.orchestra.run.smithi098.stdout:945 writesame 0xde322ce thru 0xde40eae data_size 0xd(0xebe1 bytes) 2024-09-16T13:17:08.309 INFO:teuthology.orchestra.run.smithi098.stdout:948 write 0x78985c6 thru 0x789a5e7 (0x2022 bytes) 2024-09-16T13:17:08.320 INFO:teuthology.orchestra.run.smithi098.stdout:949 trunc from 0xde40eaf to 0xa80ca33 2024-09-16T13:17:08.334 INFO:teuthology.orchestra.run.smithi098.stdout:950 trunc from 0xa80ca33 to 0x79e8702 2024-09-16T13:17:08.355 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:08.350+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:08.367 INFO:teuthology.orchestra.run.smithi098.stdout:951 write 0xb2207f2 thru 0xb22a3a1 (0x9bb0 bytes) 2024-09-16T13:17:08.375 INFO:teuthology.orchestra.run.smithi098.stdout:952 writesame 0x75ecd56 thru 0x75ed3b4 data_size 0x1(0x65f bytes) 2024-09-16T13:17:08.383 INFO:teuthology.orchestra.run.smithi098.stdout:953 punch from 0xa3eaca9 to 0xa3f9bde, (0xef35 bytes) 2024-09-16T13:17:08.383 INFO:teuthology.orchestra.run.smithi098.stdout:954 write 0x4a8b03f thru 0x4a925f5 (0x75b7 bytes) 2024-09-16T13:17:08.390 INFO:teuthology.orchestra.run.smithi098.stdout:955 punch from 0x4e46541 to 0x4e4b884, (0x5343 bytes) 2024-09-16T13:17:08.397 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:08.397 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:08.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:08.397 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:08.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.397 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:9 3.1a 3:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T13:17:08.397 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:54.299570+0000", 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.090449920999999, 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070841239999999998, 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299570+0000", 2024-09-16T13:17:08.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299570+0000", 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299573+0000", 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3170000000000002e-06 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299587+0000", 2024-09-16T13:17:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4389999999999999e-05 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299624+0000", 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6944999999999999e-05 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299645+0000", 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0947000000000001e-05 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299793+0000", 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014779800000000001 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.299805+0000", 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2248e-05 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.303287+0000", 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034814350000000002 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.303461+0000", 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000174234 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.303611+0000", 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000150605 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.305822+0000", 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022106589999999998 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.306554+0000", 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073181600000000004 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.306608+0000", 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3946999999999997e-05 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:54.306654+0000", 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5782999999999999e-05 2024-09-16T13:17:08.403 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1852 3.14 3:2d4ed96d:::rbd_data.112d2ce070e7.000000000000000e:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:02.992818+0000", 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.397201935, 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0079669909999999997, 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.404 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1852 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.992818+0000", 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.992818+0000", 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.992833+0000", 2024-09-16T13:17:08.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5255e-05 2024-09-16T13:17:08.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.992841+0000", 2024-09-16T13:17:08.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6079999999999996e-06 2024-09-16T13:17:08.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.992882+0000", 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0479000000000003e-05 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.992894+0000", 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2462e-05 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.993018+0000", 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012414899999999999 2024-09-16T13:17:08.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.993240+0000", 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022112100000000001 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.993648+0000", 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040860799999999997 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.996718+0000", 2024-09-16T13:17:08.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030697720000000001 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.000646+0000", 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039281259999999997 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.000725+0000", 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9215000000000006e-05 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.000784+0000", 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9196e-05 2024-09-16T13:17:08.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1866 3.1a 3:5c93e4bc:::rbd_data.112d2ce070e7.0000000000000018:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:02.998165+0000", 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3918549120000003, 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0083198520000000008, 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1866 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.998165+0000", 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.998165+0000", 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.998166+0000", 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4130000000000001e-06 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.998173+0000", 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2660000000000004e-06 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.998189+0000", 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.575e-05 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.998197+0000", 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4060000000000005e-06 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.998268+0000", 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0807000000000002e-05 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.998373+0000", 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104633 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.998616+0000", 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024273299999999999 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002088+0000", 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034720290000000002 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.006401+0000", 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043131589999999996 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.006457+0000", 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6328e-05 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.006484+0000", 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7328e-05 2024-09-16T13:17:08.415 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:1919 3.f e31/23 3:f419b97f:::rbd_object_map.112d2ce070e7:head v 31'27, mlcod=31'22)", 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.036162+0000", 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3538571470000003, 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077875540000000004, 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.036162+0000", 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.036162+0000", 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.036163+0000", 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7600000000000006e-07 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.036175+0000", 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1406000000000001e-05 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.036197+0000", 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2767000000000001e-05 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.036208+0000", 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0937999999999999e-05 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.036278+0000", 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9777000000000006e-05 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.036303+0000", 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4981999999999999e-05 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.043921+0000", 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0076175770000000004 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.043950+0000", 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9130999999999999e-05 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2054 3.1b 3:daad93db:::rbd_data.112dbfd5d823.000000000000024f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.551696+0000", 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8383232329999997, 2024-09-16T13:17:08.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063101850000000003, 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2054 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551696+0000", 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551696+0000", 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551698+0000", 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1399999999999998e-06 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551708+0000", 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7000000000000003e-06 2024-09-16T13:17:08.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551753+0000", 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4648999999999999e-05 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551771+0000", 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8453000000000001e-05 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552055+0000", 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028369400000000003 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552345+0000", 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000290616 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.558006+0000", 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056609329999999999 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2056 3.3 3:c55a8cd0:::rbd_data.112dbfd5d823.0000000000000251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.424 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.551925+0000", 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8380944780000004, 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063026929999999998, 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2056 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551925+0000", 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551925+0000", 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551927+0000", 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5799999999999999e-06 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551934+0000", 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0899999999999999e-06 2024-09-16T13:17:08.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551986+0000", 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2292e-05 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.551998+0000", 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.222e-05 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552880+0000", 2024-09-16T13:17:08.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00088203700000000001 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.553177+0000", 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029679700000000001 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.558228+0000", 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050506769999999999 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2060 3.1b 3:dd044d38:::rbd_data.112dbfd5d823.0000000000000255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.552043+0000", 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8379760540000003, 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0062887189999999999, 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2060 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552043+0000", 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552043+0000", 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552045+0000", 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.33e-06 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552051+0000", 2024-09-16T13:17:08.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2230000000000004e-06 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552070+0000", 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8680000000000001e-05 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552081+0000", 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.115e-05 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552971+0000", 2024-09-16T13:17:08.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00089031600000000005 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.553257+0000", 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028579099999999997 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.558332+0000", 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050752289999999997 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2063 3.1b 3:dae7cebe:::rbd_object_map.112dbfd5d823:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.552101+0000", 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8379189150000004, 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0066362210000000003, 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2063 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552101+0000", 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552101+0000", 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552102+0000", 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6500000000000001e-06 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552112+0000", 2024-09-16T13:17:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3470000000000005e-06 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552131+0000", 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9329000000000001e-05 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552141+0000", 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0003000000000001e-05 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.554131+0000", 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019899119999999999 2024-09-16T13:17:08.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.554338+0000", 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020706199999999999 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.555266+0000", 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092793300000000003 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.558391+0000", 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003124805 2024-09-16T13:17:08.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.558586+0000", 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000195105 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.558652+0000", 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6390999999999995e-05 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.558737+0000", 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4684000000000004e-05 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2064 3.1b 3:dae7cebe:::rbd_object_map.112dbfd5d823:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.552152+0000", 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8378670929999998, 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072872659999999997, 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2064 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552152+0000", 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552152+0000", 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552154+0000", 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.917e-06 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552162+0000", 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4699999999999996e-06 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552176+0000", 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4029000000000001e-05 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.552181+0000", 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5770000000000001e-06 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.555981+0000", 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038001229999999999 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.556199+0000", 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000217424 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.556889+0000", 2024-09-16T13:17:08.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00069012399999999997 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.558432+0000", 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001542652 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.559328+0000", 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00089605200000000005 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.559396+0000", 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7800999999999997e-05 2024-09-16T13:17:08.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.559440+0000", 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4097000000000001e-05 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2276 3.1b 3:dae7cebe:::rbd_object_map.112dbfd5d823:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.739267+0000", 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6507523239999999, 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006941076, 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.443 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2276 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.739267+0000", 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.739267+0000", 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.739269+0000", 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6750000000000001e-06 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.739278+0000", 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7180000000000002e-06 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.739296+0000", 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8938000000000001e-05 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.739322+0000", 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5748000000000001e-05 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.741835+0000", 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025123789999999999 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.741972+0000", 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013754800000000001 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.742394+0000", 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042234 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.745032+0000", 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026376889999999999 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.746123+0000", 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010908510000000001 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.746171+0000", 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.833e-05 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.746208+0000", 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6860000000000003e-05 2024-09-16T13:17:08.448 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2287 3.1b 3:dae7cebe:::rbd_object_map.112dbfd5d823:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.748392+0000", 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6416277109999999, 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087598720000000001, 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2287 2024-09-16T13:17:08.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.748392+0000", 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.748392+0000", 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.748393+0000", 2024-09-16T13:17:08.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.195e-06 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.748400+0000", 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5120000000000002e-06 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.748415+0000", 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4980999999999999e-05 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.748451+0000", 2024-09-16T13:17:08.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5862000000000003e-05 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.749577+0000", 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001126054 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.749649+0000", 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1180000000000006e-05 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.749991+0000", 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034286499999999998 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755237+0000", 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0052455499999999999 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757074+0000", 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018375290000000001 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757127+0000", 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2806999999999998e-05 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757152+0000", 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4337000000000001e-05 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2290 3.1b 3:dae7cebe:::rbd_object_map.112dbfd5d823:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.749998+0000", 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6400215380000001, 2024-09-16T13:17:08.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010137386, 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2290 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.749998+0000", 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.749998+0000", 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.749999+0000", 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0899999999999999e-06 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750006+0000", 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6239999999999996e-06 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750021+0000", 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4921e-05 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750038+0000", 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7927999999999998e-05 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750712+0000", 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00067389299999999995 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750787+0000", 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4746000000000006e-05 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.751196+0000", 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040918200000000001 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757485+0000", 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0062884229999999996 2024-09-16T13:17:08.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760050+0000", 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025649309999999999 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760098+0000", 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8658000000000001e-05 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760135+0000", 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6990000000000003e-05 2024-09-16T13:17:08.459 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2291 3.9 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.750889+0000", 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6391301800000004, 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0092448800000000005, 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.460 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2291 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750889+0000", 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750889+0000", 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750890+0000", 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0860000000000001e-06 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750898+0000", 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.469e-06 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750913+0000", 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5546999999999999e-05 2024-09-16T13:17:08.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750927+0000", 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3986e-05 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.750983+0000", 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5782000000000002e-05 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.751235+0000", 2024-09-16T13:17:08.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025181900000000002 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.751431+0000", 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000195639 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752508+0000", 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001077192 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760006+0000", 2024-09-16T13:17:08.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074978639999999999 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760094+0000", 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8075e-05 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760134+0000", 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0420999999999998e-05 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2292 3.1b 3:dae7cebe:::rbd_object_map.112dbfd5d823:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.751317+0000", 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6387022619999998, 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0090626549999999993, 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2292 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.751317+0000", 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.751317+0000", 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.751318+0000", 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.032e-06 2024-09-16T13:17:08.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.751324+0000", 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6949999999999997e-06 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.751337+0000", 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2931999999999999e-05 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.751344+0000", 2024-09-16T13:17:08.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3950000000000003e-06 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752045+0000", 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00070070699999999996 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752149+0000", 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010434599999999999 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752516+0000", 2024-09-16T13:17:08.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036707999999999999 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757543+0000", 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050267799999999998 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760150+0000", 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002606784 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760347+0000", 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000196677 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760380+0000", 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3226999999999999e-05 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2293 3.1b 3:dae7cebe:::rbd_object_map.112dbfd5d823:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.753165+0000", 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6368544099999998, 2024-09-16T13:17:08.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072754680000000002, 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2293 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753165+0000", 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753165+0000", 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753167+0000", 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5460000000000001e-06 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753172+0000", 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6230000000000002e-06 2024-09-16T13:17:08.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753188+0000", 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5387999999999999e-05 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753194+0000", 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9610000000000002e-06 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753257+0000", 2024-09-16T13:17:08.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3508000000000002e-05 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753361+0000", 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010388 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753780+0000", 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041886099999999997 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757606+0000", 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003826248 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760388+0000", 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027824500000000001 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760417+0000", 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8853e-05 2024-09-16T13:17:08.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760440+0000", 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.315e-05 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2295 3.1b 3:dae7cebe:::rbd_object_map.112dbfd5d823:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.753994+0000", 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6360255840000004, 2024-09-16T13:17:08.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0066178069999999999, 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2295 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753994+0000", 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753994+0000", 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753995+0000", 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8400000000000002e-07 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754000+0000", 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9450000000000001e-06 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754027+0000", 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6959999999999999e-05 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754033+0000", 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5069999999999997e-06 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754337+0000", 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030359400000000002 2024-09-16T13:17:08.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754446+0000", 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0001089 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754800+0000", 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035421500000000001 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760447+0000", 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056474259999999997 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760516+0000", 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8845999999999998e-05 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760565+0000", 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8517000000000003e-05 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760612+0000", 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6913000000000003e-05 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2432 3.9 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.004469+0000", 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3855506630000001, 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073663940000000001, 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.483 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2432 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.004469+0000", 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.004469+0000", 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.004471+0000", 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.733e-06 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.004477+0000", 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.004492+0000", 2024-09-16T13:17:08.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.49e-05 2024-09-16T13:17:08.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.004505+0000", 2024-09-16T13:17:08.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3648999999999999e-05 2024-09-16T13:17:08.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.006822+0000", 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023165569999999999 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.007047+0000", 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022479100000000001 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.007366+0000", 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031976999999999998 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.010210+0000", 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028437919999999999 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.011741+0000", 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015305200000000001 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.011801+0000", 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0738e-05 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.011835+0000", 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3772000000000001e-05 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2943 3.1b 3:dae7cebe:::rbd_object_map.112dbfd5d823:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.849394+0000", 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.5406256530000002, 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063117770000000002, 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.489 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2943 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.849394+0000", 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.849394+0000", 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.849395+0000", 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.046e-06 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.849399+0000", 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2799999999999997e-06 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.849412+0000", 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3174e-05 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.849418+0000", 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9880000000000001e-06 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.851381+0000", 2024-09-16T13:17:08.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019622989999999998 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.851636+0000", 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025509099999999999 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.852439+0000", 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080324499999999996 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.854999+0000", 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025598729999999998 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.855496+0000", 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049763200000000002 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.855613+0000", 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117014 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.855706+0000", 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2134999999999998e-05 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:5174 3.a e34/23 3:548e7ca7:::rbd_data.112d64676e7f.00000000000002f3:head v 34'35, mlcod=34'33)", 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:07.865595+0000", 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.52442485800000005, 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0085036269999999997, 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.495 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.865595+0000", 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.865595+0000", 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.865596+0000", 2024-09-16T13:17:08.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7129999999999999e-06 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.865611+0000", 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4826e-05 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.865634+0000", 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2917000000000001e-05 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.865648+0000", 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4346e-05 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.865755+0000", 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106628 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.865798+0000", 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2589000000000002e-05 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.874033+0000", 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0082355229999999998 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.874098+0000", 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5085000000000005e-05 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:5180 3.1b 3:dff3f97d:::rbd_data.112d64676e7f.00000000000002f9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:07.866693+0000", 2024-09-16T13:17:08.499 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.52332679299999996, 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086983249999999998, 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 5180 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.866693+0000", 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.866693+0000", 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.866695+0000", 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.34e-06 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.866707+0000", 2024-09-16T13:17:08.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2051e-05 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.866736+0000", 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8717e-05 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.866754+0000", 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7971000000000001e-05 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.866915+0000", 2024-09-16T13:17:08.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016128 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.867103+0000", 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018788600000000001 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.867355+0000", 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025183699999999998 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.869345+0000", 2024-09-16T13:17:08.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019903289999999999 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.875288+0000", 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0059434420000000002 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.875344+0000", 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5447e-05 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.875391+0000", 2024-09-16T13:17:08.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7024999999999998e-05 2024-09-16T13:17:08.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.505 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.505 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.505 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:08.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:958 write 0xdc163e5 thru 0xdc22769 (0xc385 bytes) 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:959 write 0x1630357 thru 0x163e8c2 (0xe56c bytes) 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:961 read 0x21191bf thru 0x2123e1f (0xac61 bytes) 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:964 write 0x9783e5b thru 0x978ef8d (0xb133 bytes) 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:965 write 0x54a8a4f thru 0x54b6927 (0xded9 bytes) 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:966 read 0x14bf047 thru 0x14bfb82 (0xb3c bytes) 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:967 trunc from 0xdc2276a to 0xe3c8f8c 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:968 read 0x303965 thru 0x307d0c (0x43a8 bytes) 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:970 write 0x73da99d thru 0x73e4c1e (0xa282 bytes) 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:971 trunc from 0xe3c8f8c to 0xeacb457 2024-09-16T13:17:08.509 INFO:teuthology.orchestra.run.smithi098.stdout:972 punch from 0x3298573 to 0x329b657, (0x30e4 bytes) 2024-09-16T13:17:08.510 INFO:teuthology.orchestra.run.smithi098.stdout:973 writesame 0x9f08a95 thru 0x9f09757 data_size 0xb(0xcc3 bytes) 2024-09-16T13:17:08.510 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:08.510 INFO:teuthology.orchestra.run.smithi098.stdout:truncating to largest ever: 0xee24112 2024-09-16T13:17:08.510 INFO:teuthology.orchestra.run.smithi098.stdout:976 trunc from 0xeacb457 to 0xee24112 2024-09-16T13:17:08.510 INFO:teuthology.orchestra.run.smithi098.stdout:977 punch from 0x43106a to 0x434296, (0x322c bytes) 2024-09-16T13:17:08.510 INFO:teuthology.orchestra.run.smithi098.stdout:978 write 0x97ea3de thru 0x97f4823 (0xa446 bytes) 2024-09-16T13:17:08.510 INFO:teuthology.orchestra.run.smithi098.stdout:979 writesame 0x35907ca thru 0x3597521 data_size 0x4(0x6d58 bytes) 2024-09-16T13:17:08.510 INFO:teuthology.orchestra.run.smithi098.stdout:980 punch from 0xdb9196c to 0xdb91c89, (0x31d bytes) 2024-09-16T13:17:08.510 INFO:teuthology.orchestra.run.smithi098.stdout:981 clone 5 order 18 su 131072 sc 3 2024-09-16T13:17:08.510 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:08.500+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:08.607 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:08.602+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:08.620 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:08.620 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:08.621 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:08.621 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:08.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:08.708 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:08.700+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:08.734 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:08.734 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:08.734 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:08.734 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:08.734 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:08.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:08.809 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:08.804+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:08.839 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:08.839 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:08.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:08.839 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2520 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8083475150000003, 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012988767, 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2520 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.840 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024618+0000", 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0199999999999998e-07 2024-09-16T13:17:08.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024620+0000", 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1150000000000001e-06 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024624+0000", 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.676e-06 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024626+0000", 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6730000000000001e-06 2024-09-16T13:17:08.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037443+0000", 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012816869 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037522+0000", 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9060000000000005e-05 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037606+0000", 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4072000000000004e-05 2024-09-16T13:17:08.843 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2521 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2523136~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.808336143, 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013552781999999999, 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.844 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2521 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024630+0000", 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1629999999999999e-06 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024635+0000", 2024-09-16T13:17:08.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5050000000000001e-06 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024637+0000", 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4040000000000002e-06 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037635+0000", 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012997754 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037729+0000", 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4043999999999996e-05 2024-09-16T13:17:08.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038182+0000", 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045262500000000001 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2522 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3473408~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8083243199999997, 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013780114, 2024-09-16T13:17:08.848 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2522 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8999999999999998e-07 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024642+0000", 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3230000000000001e-06 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024646+0000", 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024648+0000", 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4849999999999999e-06 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038251+0000", 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013602822000000001 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038335+0000", 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4215999999999995e-05 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038421+0000", 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.543e-05 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.852 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2523 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8083113749999997, 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014015147, 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.854 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2523 2024-09-16T13:17:08.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.854 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:08.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024656+0000", 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.131e-06 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024660+0000", 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6990000000000001e-06 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024663+0000", 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0290000000000001e-06 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038456+0000", 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013793481 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038549+0000", 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2608000000000005e-05 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038669+0000", 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011982800000000001 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.857 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2524 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8082978010000001, 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014381392, 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2524 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024668+0000", 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8400000000000003e-07 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024670+0000", 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.849e-06 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024675+0000", 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.451e-06 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024678+0000", 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8739999999999999e-06 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038723+0000", 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014044562999999999 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038811+0000", 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039049+0000", 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023761100000000001 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.861 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2525 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 557056~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8082634540000004, 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014602618, 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.862 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2525 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024705+0000", 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4650000000000001e-06 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024711+0000", 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024716+0000", 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1710000000000004e-06 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039096+0000", 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014380184000000001 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039187+0000", 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1065999999999999e-05 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039304+0000", 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117333 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2526 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1507328~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8082452849999999, 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014823846, 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.866 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2526 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024722+0000", 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024729+0000", 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024733+0000", 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039337+0000", 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014604026000000001 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039427+0000", 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0437000000000002e-05 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039544+0000", 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116234 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2527 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1605632~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:08.870 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8082275220000001, 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015041116, 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2527 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024738+0000", 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024746+0000", 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2130000000000008e-06 2024-09-16T13:17:08.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024753+0000", 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024758+0000", 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7670000000000003e-06 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:17:08.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014820533 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039670+0000", 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1825999999999998e-05 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039779+0000", 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108424 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2528 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8082020840000004, 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015245242000000001, 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2528 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024765+0000", 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1040000000000001e-06 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024772+0000", 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024776+0000", 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039810+0000", 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015034194000000001 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039897+0000", 2024-09-16T13:17:08.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6864e-05 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040008+0000", 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110802 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2529 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2555904~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8081829650000003, 2024-09-16T13:17:08.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015471574, 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2529 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:08.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024785+0000", 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5969999999999999e-06 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024800+0000", 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4681999999999999e-05 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024806+0000", 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040039+0000", 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015233007999999999 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040143+0000", 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010322799999999999 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040254+0000", 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011092899999999999 2024-09-16T13:17:08.883 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2530 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8081526849999996, 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015667606000000001, 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.884 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2530 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024813+0000", 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024815+0000", 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5160000000000001e-06 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024827+0000", 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1915e-05 2024-09-16T13:17:08.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024834+0000", 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5180000000000001e-06 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040286+0000", 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015452565999999999 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040375+0000", 2024-09-16T13:17:08.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040480+0000", 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105388 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2531 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3538944~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:08.888 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8081254700000002, 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015867612999999999, 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2531 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024843+0000", 2024-09-16T13:17:08.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6960000000000001e-06 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024856+0000", 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-05 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024863+0000", 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6069999999999996e-06 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040513+0000", 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650662999999999 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040599+0000", 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5439999999999995e-05 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040707+0000", 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108461 2024-09-16T13:17:08.892 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2532 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8080949650000004, 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061912000000001, 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.893 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2532 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024871+0000", 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024874+0000", 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024881+0000", 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0570000000000004e-06 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024886+0000", 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040742+0000", 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015856808 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040824+0000", 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040932+0000", 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108378 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2533 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8080744400000004, 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016272282999999998, 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.897 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2533 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:08.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024893+0000", 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419999999999999e-06 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024900+0000", 2024-09-16T13:17:08.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024903+0000", 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040965+0000", 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061565999999999 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041052+0000", 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7213000000000007e-05 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041163+0000", 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011066899999999999 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2534 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3604480~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:08.901 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8080559090000001, 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016482746999999999, 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2534 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024910+0000", 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1299999999999999e-07 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024912+0000", 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024920+0000", 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024924+0000", 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.036e-06 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041197+0000", 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016273242 2024-09-16T13:17:08.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041282+0000", 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5013999999999996e-05 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041392+0000", 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109567 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2535 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8080352470000003, 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016691902000000002, 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2535 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7599999999999997e-07 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024932+0000", 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024949+0000", 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7053000000000001e-05 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024961+0000", 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041425+0000", 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016463851000000002 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041515+0000", 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9549999999999995e-05 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041622+0000", 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107132 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.909 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2536 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1638400~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8079974410000004, 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016880869999999999, 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2536 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.910 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:08.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024971+0000", 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.615e-06 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024988+0000", 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6912999999999999e-05 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024993+0000", 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041653+0000", 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016660245000000001 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041743+0000", 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0099999999999995e-05 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041848+0000", 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105404 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2537 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1671168~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8079666880000005, 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017075346000000002, 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.914 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2537 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:08.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024999+0000", 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025002+0000", 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:17:08.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025009+0000", 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025013+0000", 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1840000000000001e-06 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041886+0000", 2024-09-16T13:17:08.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016873249999999999 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041971+0000", 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4487000000000006e-05 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042074+0000", 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102847 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.918 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2538 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2621440~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8079465169999995, 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017314057000000001, 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2538 2024-09-16T13:17:08.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025019+0000", 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025021+0000", 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025033+0000", 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1985e-05 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025043+0000", 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3810000000000005e-06 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042107+0000", 2024-09-16T13:17:08.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017064471000000001 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042203+0000", 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5810999999999995e-05 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042333+0000", 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012956099999999999 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.923 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2539 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8079187890000004, 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017500180000000001, 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:08.924 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2539 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:08.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025047+0000", 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025053+0000", 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025058+0000", 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-16T13:17:08.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025061+0000", 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.072e-06 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042365+0000", 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017304885999999998 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042458+0000", 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2991000000000005e-05 2024-09-16T13:17:08.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:08.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:08.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:08.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042546+0000", 2024-09-16T13:17:08.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7904000000000004e-05 2024-09-16T13:17:08.928 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.928 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.928 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.928 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:08.928 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:08.928 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:08.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:08.930 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:08.922+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:09.031 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:09.024+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:09.034 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:09.035 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:09.035 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:09.035 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:09.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:set 3.1c pg_upmap mapping to [6,1] 2024-09-16T13:17:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:17:09.132 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:09.124+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:09.138 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:09.138 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:09.138 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:09.138 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:09.139 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:09.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:09.233 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:09.226+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:09.247 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:09.247 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:09.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:09.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:09.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.247 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1022 3.10 3:0c4f3e94:::rbd_data.112de98401a1.000000000000002d:head [set-alloc-hint object_size 4194304 write_size 4194304,write 830679~9355] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.732415+0000", 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.50708045, 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040114139999999996, 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1022 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732415+0000", 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732415+0000", 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732418+0000", 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2770000000000001e-06 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732942+0000", 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052370100000000001 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732986+0000", 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4079000000000003e-05 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733002+0000", 2024-09-16T13:17:09.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5716999999999999e-05 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733137+0000", 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013541700000000001 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733389+0000", 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025132399999999998 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:09.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733884+0000", 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049490199999999997 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736097+0000", 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002213338 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736272+0000", 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017514900000000001 2024-09-16T13:17:09.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736363+0000", 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1467999999999999e-05 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736427+0000", 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3041999999999997e-05 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.253 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1411 3.a 3:535f7855:::rbd_data.112de98401a1.0000000000000012:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3635406~15506] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:59.845644+0000", 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.3938515099999993, 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033395920000000002, 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1411 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.845644+0000", 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.845644+0000", 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.845647+0000", 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4000000000000001e-06 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.845698+0000", 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1014999999999999e-05 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.845735+0000", 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6829999999999998e-05 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.845750+0000", 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4439e-05 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.845894+0000", 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144134 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.846131+0000", 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023714499999999999 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.846569+0000", 2024-09-16T13:17:09.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00043791300000000001 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.848706+0000", 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021373659999999999 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.848777+0000", 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0347999999999998e-05 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.848876+0000", 2024-09-16T13:17:09.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9784000000000006e-05 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:59.848984+0000", 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107218 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1482 3.a 3:535f7855:::rbd_data.112de98401a1.0000000000000012:head [truncate 2391135] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:00.072537+0000", 2024-09-16T13:17:09.259 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.1669589679999994, 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069626239999999997, 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1482 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072537+0000", 2024-09-16T13:17:09.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072537+0000", 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072539+0000", 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7829999999999999e-06 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072551+0000", 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1249e-05 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072588+0000", 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7230000000000001e-05 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072601+0000", 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3512999999999999e-05 2024-09-16T13:17:09.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072737+0000", 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000135387 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072747+0000", 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0645999999999999e-05 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.076771+0000", 2024-09-16T13:17:09.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040232200000000001 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.076893+0000", 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122731 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.077095+0000", 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020168900000000001 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.078550+0000", 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001454711 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.079360+0000", 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00081076499999999997 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.079438+0000", 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7879000000000004e-05 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.079499+0000", 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820999999999999e-05 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1516 3.a 3:535f7855:::rbd_data.112de98401a1.0000000000000012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:00.142385+0000", 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0971104240000003, 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031490870000000001, 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.266 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1516 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.142385+0000", 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.142385+0000", 2024-09-16T13:17:09.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.142388+0000", 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8270000000000002e-06 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.142401+0000", 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.311e-05 2024-09-16T13:17:09.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.142442+0000", 2024-09-16T13:17:09.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1365000000000001e-05 2024-09-16T13:17:09.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.142461+0000", 2024-09-16T13:17:09.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8110000000000001e-05 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.142630+0000", 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000169366 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.142899+0000", 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026927400000000001 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.143258+0000", 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035883600000000002 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.144516+0000", 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012579749999999999 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.145350+0000", 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00083368500000000002 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.145433+0000", 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3764999999999995e-05 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.145534+0000", 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010077399999999999 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.272 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1835 3.a 3:535385f0:::rbd_children:head [call rbd.add_child in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:02.969751+0000", 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.2697446770000003, 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067434870000000003, 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1835 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969751+0000", 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969751+0000", 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969753+0000", 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4049999999999998e-06 2024-09-16T13:17:09.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969763+0000", 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0067e-05 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969790+0000", 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6842000000000002e-05 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969799+0000", 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8789999999999992e-06 2024-09-16T13:17:09.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969867+0000", 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8363999999999998e-05 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969874+0000", 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4119999999999998e-06 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.972618+0000", 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027437809999999998 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.972799+0000", 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018139799999999999 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:09.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.973042+0000", 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024340899999999999 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.976365+0000", 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033225809999999998 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.976390+0000", 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5364e-05 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.976452+0000", 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1464000000000006e-05 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.976494+0000", 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2521000000000002e-05 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.279 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1868 3.10 3:0bfff005:::rbd_data.112d2ce070e7.0000000000000019:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:02.999819+0000", 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.239676319, 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042038400000000004, 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1868 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999819+0000", 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999819+0000", 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999821+0000", 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999830+0000", 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3350000000000007e-06 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999853+0000", 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3445999999999999e-05 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999863+0000", 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0278000000000001e-05 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999955+0000", 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1396000000000004e-05 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.000109+0000", 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015418499999999999 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.000429+0000", 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032062300000000002 2024-09-16T13:17:09.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.003908+0000", 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034785160000000001 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.003941+0000", 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2546999999999999e-05 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.003986+0000", 2024-09-16T13:17:09.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4990999999999997e-05 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.004023+0000", 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7531999999999999e-05 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1874 3.a 3:55586239:::rbd_data.112d2ce070e7.000000000000001d:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.002278+0000", 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.2372176709999998, 2024-09-16T13:17:09.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048428020000000002, 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1874 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002278+0000", 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002278+0000", 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002279+0000", 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2529999999999999e-06 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002285+0000", 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7080000000000002e-06 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002303+0000", 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8026999999999999e-05 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002310+0000", 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7750000000000004e-06 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002374+0000", 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4634000000000005e-05 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002534+0000", 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016008999999999999 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.007121+0000", 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045863149999999997 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1886 3.a 3:55586239:::rbd_data.112d2ce070e7.000000000000001d:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.008537+0000", 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.2309589179999998, 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006065801, 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1886 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008537+0000", 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008537+0000", 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008538+0000", 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.128e-06 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008544+0000", 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2650000000000002e-06 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008556+0000", 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2132e-05 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008562+0000", 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1129999999999997e-06 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008653+0000", 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1044000000000004e-05 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008746+0000", 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2497999999999994e-05 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008930+0000", 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018463900000000001 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012488+0000", 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035579790000000002 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014501+0000", 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020121340000000001 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014559+0000", 2024-09-16T13:17:09.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7967000000000003e-05 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014602+0000", 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3902000000000001e-05 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1888 3.10 3:0b516a8b:::rbd_data.112d2ce070e7.0000000000000025:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.008578+0000", 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.230917786, 2024-09-16T13:17:09.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034403580000000001, 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1888 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008578+0000", 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008578+0000", 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008578+0000", 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6000000000000003e-07 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008582+0000", 2024-09-16T13:17:09.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3960000000000002e-06 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008595+0000", 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2696e-05 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008602+0000", 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0910000000000004e-06 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008747+0000", 2024-09-16T13:17:09.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014532600000000001 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008874+0000", 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012695499999999999 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012018+0000", 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031442340000000001 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.299 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:1894 3.1e e31/23 3:7c25ca39:::rbd_data.112d2ce070e7.0000000000000024:head v 31'2, mlcod=0'0)", 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.010400+0000", 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.2290955459999999, 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041542200000000001, 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010400+0000", 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010400+0000", 2024-09-16T13:17:09.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010401+0000", 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6899999999999997e-07 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010406+0000", 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010416+0000", 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6679999999999996e-06 2024-09-16T13:17:09.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010420+0000", 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010493+0000", 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3345999999999999e-05 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010525+0000", 2024-09-16T13:17:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1677000000000002e-05 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014522+0000", 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039971540000000002 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014554+0000", 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1906000000000003e-05 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1897 3.10 3:0b516a8b:::rbd_data.112d2ce070e7.0000000000000025:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.012651+0000", 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.2268443449999999, 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046366310000000004, 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1897 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012651+0000", 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012651+0000", 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012652+0000", 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.102e-06 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012659+0000", 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8580000000000002e-06 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012677+0000", 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7880000000000002e-05 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012684+0000", 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4159999999999998e-06 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012729+0000", 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4883999999999999e-05 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012815+0000", 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6019000000000004e-05 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.013058+0000", 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024225399999999999 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014458+0000", 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001400016 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.017180+0000", 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027226160000000002 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.017232+0000", 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1452000000000001e-05 2024-09-16T13:17:09.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.017288+0000", 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6134000000000002e-05 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1925 3.a 3:574306b8:::rbd_data.112d2ce070e7.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,truncate 1179648] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.042069+0000", 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.1974270000000002, 2024-09-16T13:17:09.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045267179999999999, 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1925 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042069+0000", 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042069+0000", 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042071+0000", 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8200000000000001e-06 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042081+0000", 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8269999999999993e-06 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042106+0000", 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4858000000000002e-05 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042115+0000", 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9409999999999996e-06 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042182+0000", 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6631000000000006e-05 2024-09-16T13:17:09.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042307+0000", 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012550699999999999 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042451+0000", 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014376300000000001 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.043788+0000", 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001337095 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.046474+0000", 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026856419999999998 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.046546+0000", 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2175000000000006e-05 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.046595+0000", 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9459000000000002e-05 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2297 3.10 3:0ec99851:::rbd_data.112dbfd5d823.0000000000000036:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.757373+0000", 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.4821226699999999, 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002923873, 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.315 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2297 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757373+0000", 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757373+0000", 2024-09-16T13:17:09.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757375+0000", 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.475e-06 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757388+0000", 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2564e-05 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757413+0000", 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5048000000000001e-05 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757423+0000", 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7799999999999995e-06 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757507+0000", 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4536999999999994e-05 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757689+0000", 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018175900000000001 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.757824+0000", 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013455100000000001 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.759135+0000", 2024-09-16T13:17:09.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013115309999999999 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760196+0000", 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001061083 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760252+0000", 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5699999999999999e-05 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.760297+0000", 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4845000000000002e-05 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2597 3.a 3:56dabaf7:::rbd_data.112dbfd5d823.0000000000000272:head [set-alloc-hint object_size 262144 write_size 262144,write 65536~6347] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.130681+0000", 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.1088141419999999, 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030980420000000001, 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.321 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2597 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.130681+0000", 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.130681+0000", 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.130685+0000", 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3440000000000001e-06 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.130709+0000", 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3869000000000001e-05 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.130744+0000", 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5423000000000003e-05 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.130760+0000", 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5472e-05 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.130914+0000", 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015492699999999999 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.131306+0000", 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039144800000000002 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.131716+0000", 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041007700000000001 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.133101+0000", 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013852109999999999 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.133585+0000", 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00048359199999999997 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.133669+0000", 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4450000000000006e-05 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.133779+0000", 2024-09-16T13:17:09.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110229 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2820 3.10 3:0e7d147a:::rbd_data.112dbfd5d823.0000000000000384:head [set-alloc-hint object_size 262144 write_size 262144,write 152843~6990] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.587714+0000", 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6517813859999997, 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029797209999999998, 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2820 2024-09-16T13:17:09.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.587714+0000", 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.587714+0000", 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.587717+0000", 2024-09-16T13:17:09.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8320000000000002e-06 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.587737+0000", 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0149999999999999e-05 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.587769+0000", 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2319e-05 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.587781+0000", 2024-09-16T13:17:09.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1871999999999999e-05 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.587867+0000", 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5697000000000003e-05 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.588091+0000", 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022416499999999999 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.588291+0000", 2024-09-16T13:17:09.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019988499999999999 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.589577+0000", 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001286272 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.590570+0000", 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00099248700000000001 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.590642+0000", 2024-09-16T13:17:09.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1940999999999994e-05 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.590694+0000", 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2101000000000001e-05 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:2922 3.1e e31/23 3:7ca2e887:::rbd_data.112dbfd5d823.000000000000029f:head v 31'4, mlcod=31'2)", 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.807258+0000", 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.432237089, 2024-09-16T13:17:09.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034921599999999998, 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.807258+0000", 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.807258+0000", 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.807260+0000", 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.007e-06 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.807279+0000", 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8635e-05 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.807314+0000", 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5305000000000002e-05 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.807330+0000", 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5574e-05 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.807423+0000", 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2664000000000006e-05 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.807487+0000", 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4306000000000004e-05 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.810653+0000", 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031661660000000002 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.810751+0000", 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7503000000000005e-05 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:5174 3.a 3:548e7ca7:::rbd_data.112d64676e7f.00000000000002f3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:09.336 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:07.863709+0000", 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.3757869549999999, 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010653861000000001, 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 5174 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863709+0000", 2024-09-16T13:17:09.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863709+0000", 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863711+0000", 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1399999999999998e-06 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863721+0000", 2024-09-16T13:17:09.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0727000000000001e-05 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863757+0000", 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5525999999999998e-05 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863770+0000", 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2595999999999999e-05 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863897+0000", 2024-09-16T13:17:09.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012721100000000001 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.864134+0000", 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000237144 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.864431+0000", 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029679400000000002 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.866015+0000", 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001584366 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.874278+0000", 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0082627359999999997 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.874316+0000", 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8427999999999999e-05 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.874362+0000", 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6193000000000003e-05 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:5216 3.10 3:0a27cd76:::rbd_data.112d64676e7f.000000000000023d:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:07.884031+0000", 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.3554650340000001, 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029124239999999998, 2024-09-16T13:17:09.342 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 5216 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.884031+0000", 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.884031+0000", 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.884032+0000", 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.884055+0000", 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3153e-05 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.884082+0000", 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6693e-05 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.884091+0000", 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8759999999999994e-06 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.884190+0000", 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9936e-05 2024-09-16T13:17:09.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.884465+0000", 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000274999 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.884834+0000", 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036823800000000001 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.885762+0000", 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092803199999999997 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.886842+0000", 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001080374 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.886904+0000", 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1453999999999998e-05 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.886943+0000", 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9339999999999999e-05 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:5258 3.10 3:0a27cd76:::rbd_data.112d64676e7f.000000000000023d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:07.938055+0000", 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.3014401790000001, 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029608999999999998, 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.348 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 5258 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.938055+0000", 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.938055+0000", 2024-09-16T13:17:09.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.938057+0000", 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.872e-06 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.938072+0000", 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4628e-05 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.938110+0000", 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7922e-05 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.938124+0000", 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4348999999999999e-05 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.938245+0000", 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012102199999999999 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.938358+0000", 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112725 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.938551+0000", 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000193036 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.940665+0000", 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021136380000000001 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.940876+0000", 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021116200000000001 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.940963+0000", 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7417999999999995e-05 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.941016+0000", 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3127999999999997e-05 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:5268 3.a 3:56d61984:::rbd_data.112d64676e7f.0000000000000247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:07.942752+0000", 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.296743545, 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033678079999999999, 2024-09-16T13:17:09.354 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 5268 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.942752+0000", 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:09.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.942752+0000", 2024-09-16T13:17:09.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:09.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:09.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.942754+0000", 2024-09-16T13:17:09.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3630000000000001e-06 2024-09-16T13:17:09.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:09.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.942763+0000", 2024-09-16T13:17:09.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7220000000000005e-06 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.942785+0000", 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.19e-05 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.942801+0000", 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5676999999999999e-05 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:09.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.942902+0000", 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101742 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.943064+0000", 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016160600000000001 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.944001+0000", 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093685899999999996 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.945189+0000", 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011883410000000001 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.946005+0000", 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000815863 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.946065+0000", 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0069999999999999e-05 2024-09-16T13:17:09.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:09.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:09.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:09.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.946120+0000", 2024-09-16T13:17:09.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4664999999999999e-05 2024-09-16T13:17:09.361 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.361 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.361 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.361 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:09.361 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:09.361 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:09.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:09.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:09.364 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:09.358+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:09.464 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:09.456+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:09.478 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:09.478 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:09.478 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:09.478 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:09.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:09.507 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:09.527 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:09.549 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:09.565 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:09.560+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:09.575 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:09.590 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:09.590 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:09.590 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:09.590 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:09.591 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:09.595 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:09.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:09.615 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:09.636 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:09.655 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:09.658 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:09.658 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:09.658 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:09.658 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:09.658 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:09.658 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:09.659 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:09.659 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:09.668 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:09.662+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:09.705 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:09.706 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:09.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:09.774 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:09.770+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:09.827 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:09.827 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:09.848 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:09.902 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:09.896+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:09.961 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:09.962 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:09.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:09.971 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:09.964+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:10.070 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:10.064+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:10.077 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:10.078 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:10.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:10.155 INFO:teuthology.orchestra.run.smithi098.stdout:flattening image image_client.0-clone4 2024-09-16T13:17:10.171 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:10.164+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:10.242 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:10.242 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:10.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:10.272 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:10.264+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:10.355 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:10.355 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:10.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:10.373 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:10.366+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:10.399 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #3, image image_client.0-clone3 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent4 2024-09-16T13:17:10.474 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:10.470+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:10.500 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:10.501 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:10.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:10.575 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:10.570+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:10.618 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:10.618 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:10.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:10.676 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:10.668+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:10.728 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:10.729 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:10.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:10.777 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:10.772+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:10.837 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:10.838 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:10.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:10.859 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:10.859 INFO:teuthology.orchestra.run.smithi098.stdout:984 read 0xcddac93 thru 0xcddbe08 (0x1176 bytes) 2024-09-16T13:17:10.860 INFO:teuthology.orchestra.run.smithi098.stdout:986 punch from 0xde51002 to 0xde556e4, (0x46e2 bytes) 2024-09-16T13:17:10.867 INFO:teuthology.orchestra.run.smithi098.stdout:988 read 0xeb7c3e9 thru 0xeb8c111 (0xfd29 bytes) 2024-09-16T13:17:10.867 INFO:teuthology.orchestra.run.smithi098.stdout:989 write 0xd7ff4fa thru 0xd809694 (0xa19b bytes) 2024-09-16T13:17:10.877 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:10.870+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:10.879 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:10.889 INFO:teuthology.orchestra.run.smithi098.stdout:992 trunc from 0xee24112 to 0xae82984 2024-09-16T13:17:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:993 punch from 0x78b39d6 to 0x78b81bc, (0x47e6 bytes) 2024-09-16T13:17:10.917 INFO:teuthology.orchestra.run.smithi098.stdout:994 write 0x9a14755 thru 0x9a1918b (0x4a37 bytes) 2024-09-16T13:17:10.934 INFO:teuthology.orchestra.run.smithi098.stdout:996 trunc from 0xae82984 to 0x5786f86 2024-09-16T13:17:10.956 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:10.956 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:10.963 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:10.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:10.978 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:10.974+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:10.979 INFO:teuthology.orchestra.run.smithi098.stdout:998 writesame 0x82f3225 thru 0x8302cc6 data_size 0x7(0xfaa2 bytes) 2024-09-16T13:17:10.979 INFO:teuthology.orchestra.run.smithi098.stdout:999 trunc from 0x8302cc7 to 0x852c8ba 2024-09-16T13:17:11.007 INFO:teuthology.orchestra.run.smithi098.stdout:1000 write 0xba856f7 thru 0xba8ea4a (0x9354 bytes) 2024-09-16T13:17:11.019 INFO:teuthology.orchestra.run.smithi098.stdout:1001 trunc from 0xba8ea4b to 0x9e97817 2024-09-16T13:17:11.040 INFO:teuthology.orchestra.run.smithi098.stdout:1003 read 0x8189d5c thru 0x81975fb (0xd8a0 bytes) 2024-09-16T13:17:11.040 INFO:teuthology.orchestra.run.smithi098.stdout:1004 write 0x1d6973d thru 0x1d786eb (0xefaf bytes) 2024-09-16T13:17:11.047 INFO:teuthology.orchestra.run.smithi098.stdout:1005 writesame 0x7f0b353 thru 0x7f17481 data_size 0x3f(0xc12f bytes) 2024-09-16T13:17:11.061 INFO:teuthology.orchestra.run.smithi098.stdout:1006 trunc from 0x9e97817 to 0x6a12521 2024-09-16T13:17:11.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:17:11.080 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.080 INFO:teuthology.orchestra.run.smithi098.stdout:1008 read 0x38b3983 thru 0x38b4c9b (0x1319 bytes) 2024-09-16T13:17:11.080 INFO:teuthology.orchestra.run.smithi098.stdout:1009 punch from 0x9e1e6e to 0x9ecb4b, (0xacdd bytes) 2024-09-16T13:17:11.080 INFO:teuthology.orchestra.run.smithi098.stdout:1011 writesame 0x111e5f6 thru 0x111f06b data_size 0xd(0xa76 bytes) 2024-09-16T13:17:11.080 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:11.072+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:11.083 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:11.083 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:11.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:11.105 INFO:teuthology.orchestra.run.smithi098.stdout:1013 writesame 0xb289c53 thru 0xb28cf5a data_size 0x2e(0x3308 bytes) 2024-09-16T13:17:11.105 INFO:teuthology.orchestra.run.smithi098.stdout:1014 read 0x84b95be thru 0x84c0d01 (0x7744 bytes) 2024-09-16T13:17:11.105 INFO:teuthology.orchestra.run.smithi098.stdout:1015 writesame 0x8904ab9 thru 0x8909d27 data_size 0x1(0x526f bytes) 2024-09-16T13:17:11.129 INFO:teuthology.orchestra.run.smithi098.stdout:1017 writesame 0x7b0f058 thru 0x7b1724a data_size 0xd(0x81f3 bytes) 2024-09-16T13:17:11.139 INFO:teuthology.orchestra.run.smithi098.stdout:1018 punch from 0x9b19ec8 to 0x9b1c2f8, (0x2430 bytes) 2024-09-16T13:17:11.152 INFO:teuthology.orchestra.run.smithi098.stdout:1019 trunc from 0xb28cf5b to 0x4b5a0a1 2024-09-16T13:17:11.180 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:11.174+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:11.192 INFO:teuthology.orchestra.run.smithi098.stdout:1020 write 0x51aeb91 thru 0x51b26ff (0x3b6f bytes) 2024-09-16T13:17:11.198 INFO:teuthology.orchestra.run.smithi098.stdout:1023 read 0x793753 thru 0x7a3747 (0xfff5 bytes) 2024-09-16T13:17:11.208 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:11.208 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:11.208 INFO:teuthology.orchestra.run.smithi098.stdout:1024 write 0x585f291 thru 0x5861f8e (0x2cfe bytes) 2024-09-16T13:17:11.217 INFO:teuthology.orchestra.run.smithi098.stdout:1026 read 0x24686a1 thru 0x2476f58 (0xe8b8 bytes) 2024-09-16T13:17:11.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:11.244 INFO:teuthology.orchestra.run.smithi098.stdout:1027 write 0xe975c58 thru 0xe9843d6 (0xe77f bytes) 2024-09-16T13:17:11.265 INFO:teuthology.orchestra.run.smithi098.stdout:1028 trunc from 0xe9843d7 to 0x7ad2126 2024-09-16T13:17:11.280 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:11.272+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:11.286 INFO:teuthology.orchestra.run.smithi098.stdout:1029 writesame 0x7958d4f thru 0x795c86f data_size 0x1(0x3b21 bytes) 2024-09-16T13:17:11.308 INFO:teuthology.orchestra.run.smithi098.stdout:1030 write 0x6e0a6e thru 0x6e32a2 (0x2835 bytes) 2024-09-16T13:17:11.316 INFO:teuthology.orchestra.run.smithi098.stdout:1031 read 0xc2e07 thru 0xcebce (0xbdc8 bytes) 2024-09-16T13:17:11.317 INFO:teuthology.orchestra.run.smithi098.stdout:1032 write 0x57dddc3 thru 0x57e9d3d (0xbf7b bytes) 2024-09-16T13:17:11.330 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:11.331 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:11.337 INFO:teuthology.orchestra.run.smithi098.stdout:1035 trunc from 0x7ad2126 to 0x6933a9 2024-09-16T13:17:11.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:11.381 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:11.374+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:11.392 INFO:teuthology.orchestra.run.smithi098.stdout:1036 writesame 0xb5d5188 thru 0xb5d94ca data_size 0x1(0x4343 bytes) 2024-09-16T13:17:11.392 INFO:teuthology.orchestra.run.smithi098.stdout:1037 punch from 0x74c49ac to 0x74cd05a, (0x86ae bytes) 2024-09-16T13:17:11.392 INFO:teuthology.orchestra.run.smithi098.stdout:1038 punch from 0x71dbdd8 to 0x71e6f78, (0xb1a0 bytes) 2024-09-16T13:17:11.392 INFO:teuthology.orchestra.run.smithi098.stdout:1039 read 0x1973b97 thru 0x197bd2e (0x8198 bytes) 2024-09-16T13:17:11.403 INFO:teuthology.orchestra.run.smithi098.stdout:1040 writesame 0xbc25b8e thru 0xbc27184 data_size 0x1(0x15f7 bytes) 2024-09-16T13:17:11.403 INFO:teuthology.orchestra.run.smithi098.stdout:1041 punch from 0x237e499 to 0x238b9c9, (0xd530 bytes) 2024-09-16T13:17:11.403 INFO:teuthology.orchestra.run.smithi098.stdout:1043 punch from 0x766d18e to 0x766d32b, (0x19d bytes) 2024-09-16T13:17:11.404 INFO:teuthology.orchestra.run.smithi098.stdout:1044 read 0xaa92dd6 thru 0xaa95291 (0x24bc bytes) 2024-09-16T13:17:11.404 INFO:teuthology.orchestra.run.smithi098.stdout:1046 read 0x2477f34 thru 0x24805cc (0x8699 bytes) 2024-09-16T13:17:11.404 INFO:teuthology.orchestra.run.smithi098.stdout:1048 read 0x2ad8c7f thru 0x2ae5467 (0xc7e9 bytes) 2024-09-16T13:17:11.404 INFO:teuthology.orchestra.run.smithi098.stdout:1050 read 0xac353dc thru 0xac41965 (0xc58a bytes) 2024-09-16T13:17:11.404 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.404 INFO:teuthology.orchestra.run.smithi098.stdout:1052 read 0x10e4dc0 thru 0x10edd61 (0x8fa2 bytes) 2024-09-16T13:17:11.404 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.424 INFO:teuthology.orchestra.run.smithi098.stdout:1057 trunc from 0xbc27185 to 0x51ba57 2024-09-16T13:17:11.439 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:11.439 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:11.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:11.455 INFO:teuthology.orchestra.run.smithi098.stdout:1059 read 0x1c1b thru 0xfec0 (0xe2a6 bytes) 2024-09-16T13:17:11.455 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.466 INFO:teuthology.orchestra.run.smithi098.stdout:1061 trunc from 0x51ba57 to 0x5cff0f8 2024-09-16T13:17:11.476 INFO:teuthology.orchestra.run.smithi098.stdout:1062 read 0x546f6e7 thru 0x547f67e (0xff98 bytes) 2024-09-16T13:17:11.476 INFO:teuthology.orchestra.run.smithi098.stdout:1065 read 0x3a2b69c thru 0x3a36479 (0xadde bytes) 2024-09-16T13:17:11.476 INFO:teuthology.orchestra.run.smithi098.stdout:1067 punch from 0x1d39151 to 0x1d3aef3, (0x1da2 bytes) 2024-09-16T13:17:11.476 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.482 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:11.476+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:11.491 INFO:teuthology.orchestra.run.smithi098.stdout:1069 trunc from 0x5cff0f8 to 0xe29bb4d 2024-09-16T13:17:11.502 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.502 INFO:teuthology.orchestra.run.smithi098.stdout:1073 write 0x9edb151 thru 0x9edbd93 (0xc43 bytes) 2024-09-16T13:17:11.508 INFO:teuthology.orchestra.run.smithi098.stdout:1074 read 0x2fba2c9 thru 0x2fc8038 (0xdd70 bytes) 2024-09-16T13:17:11.526 INFO:teuthology.orchestra.run.smithi098.stdout:1075 trunc from 0xe29bb4d to 0x451015b 2024-09-16T13:17:11.545 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.545 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.546 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:11.546 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:11.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:11.572 INFO:teuthology.orchestra.run.smithi098.stdout:1081 write 0xe392deb thru 0xe3940b1 (0x12c7 bytes) 2024-09-16T13:17:11.578 INFO:teuthology.orchestra.run.smithi098.stdout:1082 read 0x9779e75 thru 0x9783674 (0x9800 bytes) 2024-09-16T13:17:11.578 INFO:teuthology.orchestra.run.smithi098.stdout:1083 read 0x539c81b thru 0x53aa94c (0xe132 bytes) 2024-09-16T13:17:11.578 INFO:teuthology.orchestra.run.smithi098.stdout:1084 read 0x3de7fe5 thru 0x3de81ae (0x1ca bytes) 2024-09-16T13:17:11.582 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:11.578+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:11.588 INFO:teuthology.orchestra.run.smithi098.stdout:1085 write 0xe8b2b83 thru 0xe8bc70a (0x9b88 bytes) 2024-09-16T13:17:11.596 INFO:teuthology.orchestra.run.smithi098.stdout:1086 read 0x7a33791 thru 0x7a36dde (0x364e bytes) 2024-09-16T13:17:11.596 INFO:teuthology.orchestra.run.smithi098.stdout:1087 write 0xdef43e9 thru 0xdf01e41 (0xda59 bytes) 2024-09-16T13:17:11.605 INFO:teuthology.orchestra.run.smithi098.stdout:1088 read 0x98d8bc thru 0x99027b (0x29c0 bytes) 2024-09-16T13:17:11.606 INFO:teuthology.orchestra.run.smithi098.stdout:1089 read 0x2d1d277 thru 0x2d2d269 (0xfff3 bytes) 2024-09-16T13:17:11.606 INFO:teuthology.orchestra.run.smithi098.stdout:1090 write 0x6c74f51 thru 0x6c7d8e9 (0x8999 bytes) 2024-09-16T13:17:11.612 INFO:teuthology.orchestra.run.smithi098.stdout:1091 write 0x4df9c8d thru 0x4e018c8 (0x7c3c bytes) 2024-09-16T13:17:11.621 INFO:teuthology.orchestra.run.smithi098.stdout:1092 write 0x1cec32f thru 0x1cf6ee9 (0xabbb bytes) 2024-09-16T13:17:11.627 INFO:teuthology.orchestra.run.smithi098.stdout:1093 write 0xa819f57 thru 0xa81f5cf (0x5679 bytes) 2024-09-16T13:17:11.633 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.633 INFO:teuthology.orchestra.run.smithi098.stdout:1096 trunc from 0xe8bc70b to 0xe8b15fc 2024-09-16T13:17:11.657 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:11.657 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:11.659 INFO:teuthology.orchestra.run.smithi098.stdout:1097 trunc from 0xe8b15fc to 0x90ec09f 2024-09-16T13:17:11.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:11.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:11.683 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:11.678+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:11.696 INFO:teuthology.orchestra.run.smithi098.stdout:1098 writesame 0xdd1e2d5 thru 0xdd23c84 data_size 0x52(0x59b0 bytes) 2024-09-16T13:17:11.697 INFO:teuthology.orchestra.run.smithi098.stdout:1099 trunc from 0xdd23c85 to 0xe3c161c 2024-09-16T13:17:11.706 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.706 INFO:teuthology.orchestra.run.smithi098.stdout:1101 write 0x269e4ee thru 0x26a6767 (0x827a bytes) 2024-09-16T13:17:11.715 INFO:teuthology.orchestra.run.smithi098.stdout:1102 writesame 0x1584c08 thru 0x159194d data_size 0x19(0xcd46 bytes) 2024-09-16T13:17:11.725 INFO:teuthology.orchestra.run.smithi098.stdout:1103 write 0xcf02088 thru 0xcf091f4 (0x716d bytes) 2024-09-16T13:17:11.731 INFO:teuthology.orchestra.run.smithi098.stdout:1104 read 0x224d425 thru 0x2254e94 (0x7a70 bytes) 2024-09-16T13:17:11.731 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.731 INFO:teuthology.orchestra.run.smithi098.stdout:1106 write 0x5f679f8 thru 0x5f69331 (0x193a bytes) 2024-09-16T13:17:11.736 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.736 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.736 INFO:teuthology.orchestra.run.smithi098.stdout:1110 writesame 0x560d682 thru 0x5611c9c data_size 0x83(0x461b bytes) 2024-09-16T13:17:11.751 INFO:teuthology.orchestra.run.smithi098.stdout:1111 write 0xea379eb thru 0xea38e2d (0x1443 bytes) 2024-09-16T13:17:11.757 INFO:teuthology.orchestra.run.smithi098.stdout:1112 writesame 0xd3163b6 thru 0xd319691 data_size 0x54(0x32dc bytes) 2024-09-16T13:17:11.763 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:11.763 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:11.763 INFO:teuthology.orchestra.run.smithi098.stdout:1113 punch from 0x1d81a5a to 0x1d9161a, (0xfbc0 bytes) 2024-09-16T13:17:11.764 INFO:teuthology.orchestra.run.smithi098.stdout:1114 writesame 0x3881ed5 thru 0x389040f data_size 0x1f(0xe53b bytes) 2024-09-16T13:17:11.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:11.778 INFO:teuthology.orchestra.run.smithi098.stdout:1115 trunc from 0xea38e2e to 0xc11fdbf 2024-09-16T13:17:11.784 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:11.776+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:11.797 INFO:teuthology.orchestra.run.smithi098.stdout:1117 write 0xb83f3db thru 0xb848014 (0x8c3a bytes) 2024-09-16T13:17:11.807 INFO:teuthology.orchestra.run.smithi098.stdout:1118 trunc from 0xc11fdbf to 0xdb3ee08 2024-09-16T13:17:11.810 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:11.816 INFO:teuthology.orchestra.run.smithi098.stdout:1119 writesame 0xb0954f thru 0xb1102f data_size 0x1(0x7ae1 bytes) 2024-09-16T13:17:11.832 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:11.846 INFO:teuthology.orchestra.run.smithi098.stdout:1120 write 0x678b42d thru 0x6793e38 (0x8a0c bytes) 2024-09-16T13:17:11.853 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:11.865 INFO:teuthology.orchestra.run.smithi098.stdout:1122 writesame 0xe2c482d thru 0xe2d38cd data_size 0x1(0xf0a1 bytes) 2024-09-16T13:17:11.865 INFO:teuthology.orchestra.run.smithi098.stdout:1123 write 0xad99775 thru 0xad9f430 (0x5cbc bytes) 2024-09-16T13:17:11.871 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:11.873 INFO:teuthology.orchestra.run.smithi098.stdout:1128 write 0xce03c1b thru 0xce12e47 (0xf22d bytes) 2024-09-16T13:17:11.885 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:11.880+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:11.889 INFO:teuthology.orchestra.run.smithi098.stdout:1129 writesame 0xe4bddad thru 0xe4c121d data_size 0x19(0x3471 bytes) 2024-09-16T13:17:11.890 INFO:teuthology.orchestra.run.smithi098.stdout:1130 write 0xe458538 thru 0xe465088 (0xcb51 bytes) 2024-09-16T13:17:11.891 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:11.899 INFO:teuthology.orchestra.run.smithi098.stdout:1131 read 0x7608272 thru 0x76105c1 (0x8350 bytes) 2024-09-16T13:17:11.899 INFO:teuthology.orchestra.run.smithi098.stdout:1133 punch from 0x39ddd8 to 0x3a3334, (0x555c bytes) 2024-09-16T13:17:11.899 INFO:teuthology.orchestra.run.smithi098.stdout:1135 write 0x773ebbd thru 0x773f640 (0xa84 bytes) 2024-09-16T13:17:11.904 INFO:teuthology.orchestra.run.smithi098.stdout:1137 write 0xb911183 thru 0xb91d7ee (0xc66c bytes) 2024-09-16T13:17:11.910 INFO:teuthology.orchestra.run.smithi098.stdout:1138 write 0x6b51575 thru 0x6b52301 (0xd8d bytes) 2024-09-16T13:17:11.915 INFO:teuthology.orchestra.run.smithi098.stdout:1139 punch from 0xd162666 to 0xd1703a2, (0xdd3c bytes) 2024-09-16T13:17:11.915 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:11.916 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2520 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.896629892, 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012988767, 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.927 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2520 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024618+0000", 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0199999999999998e-07 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024620+0000", 2024-09-16T13:17:11.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1150000000000001e-06 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024624+0000", 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.676e-06 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024626+0000", 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6730000000000001e-06 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037443+0000", 2024-09-16T13:17:11.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012816869 2024-09-16T13:17:11.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037522+0000", 2024-09-16T13:17:11.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9060000000000005e-05 2024-09-16T13:17:11.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037606+0000", 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4072000000000004e-05 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2521 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2523136~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8966185199999996, 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013552781999999999, 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.932 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2521 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024630+0000", 2024-09-16T13:17:11.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1629999999999999e-06 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024635+0000", 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5050000000000001e-06 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024637+0000", 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4040000000000002e-06 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037635+0000", 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012997754 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037729+0000", 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4043999999999996e-05 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038182+0000", 2024-09-16T13:17:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045262500000000001 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2522 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3473408~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8966066970000002, 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013780114, 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2522 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8999999999999998e-07 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024642+0000", 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3230000000000001e-06 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024646+0000", 2024-09-16T13:17:11.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024648+0000", 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4849999999999999e-06 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038251+0000", 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013602822000000001 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038335+0000", 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4215999999999995e-05 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038421+0000", 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.543e-05 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2523 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:11.941 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8965937520000002, 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014015147, 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2523 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024656+0000", 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.131e-06 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024660+0000", 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6990000000000001e-06 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024663+0000", 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0290000000000001e-06 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038456+0000", 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013793481 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038549+0000", 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2608000000000005e-05 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038669+0000", 2024-09-16T13:17:11.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011982800000000001 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2524 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8965801779999998, 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014381392, 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2524 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024668+0000", 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8400000000000003e-07 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024670+0000", 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.849e-06 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024675+0000", 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.451e-06 2024-09-16T13:17:11.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024678+0000", 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8739999999999999e-06 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038723+0000", 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014044562999999999 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038811+0000", 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039049+0000", 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023761100000000001 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2525 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 557056~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:11.950 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8965458310000001, 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014602618, 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2525 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024705+0000", 2024-09-16T13:17:11.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4650000000000001e-06 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024711+0000", 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024716+0000", 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1710000000000004e-06 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039096+0000", 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014380184000000001 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039187+0000", 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1065999999999999e-05 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039304+0000", 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117333 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.954 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2526 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1507328~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8965276620000004, 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014823846, 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2526 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.955 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024722+0000", 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024729+0000", 2024-09-16T13:17:11.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024733+0000", 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039337+0000", 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014604026000000001 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039427+0000", 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0437000000000002e-05 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039544+0000", 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116234 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2527 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1605632~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.959 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8965098989999998, 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015041116, 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2527 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024738+0000", 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-16T13:17:11.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024746+0000", 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2130000000000008e-06 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024753+0000", 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024758+0000", 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7670000000000003e-06 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014820533 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039670+0000", 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1825999999999998e-05 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039779+0000", 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108424 2024-09-16T13:17:11.963 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2528 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.896484461, 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015245242000000001, 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.964 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2528 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024765+0000", 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1040000000000001e-06 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024772+0000", 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-16T13:17:11.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024776+0000", 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039810+0000", 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015034194000000001 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039897+0000", 2024-09-16T13:17:11.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6864e-05 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040008+0000", 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110802 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2529 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2555904~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.968 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8964653419999999, 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015471574, 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2529 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024785+0000", 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5969999999999999e-06 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024800+0000", 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4681999999999999e-05 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024806+0000", 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040039+0000", 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015233007999999999 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040143+0000", 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010322799999999999 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040254+0000", 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011092899999999999 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2530 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8964350620000001, 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015667606000000001, 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.973 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2530 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:11.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024813+0000", 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024815+0000", 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5160000000000001e-06 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024827+0000", 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1915e-05 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024834+0000", 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5180000000000001e-06 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040286+0000", 2024-09-16T13:17:11.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015452565999999999 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040375+0000", 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040480+0000", 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105388 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.977 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2531 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3538944~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8964078469999999, 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015867612999999999, 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.978 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2531 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:11.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024843+0000", 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6960000000000001e-06 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024856+0000", 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-05 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024863+0000", 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6069999999999996e-06 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040513+0000", 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650662999999999 2024-09-16T13:17:11.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040599+0000", 2024-09-16T13:17:11.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5439999999999995e-05 2024-09-16T13:17:11.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040707+0000", 2024-09-16T13:17:11.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108461 2024-09-16T13:17:11.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2532 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8963773420000001, 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061912000000001, 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.983 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2532 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024871+0000", 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024874+0000", 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024881+0000", 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0570000000000004e-06 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024886+0000", 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040742+0000", 2024-09-16T13:17:11.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015856808 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040824+0000", 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040932+0000", 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108378 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2533 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.896356817, 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016272282999999998, 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2533 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.988 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024893+0000", 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419999999999999e-06 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024900+0000", 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024903+0000", 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040965+0000", 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061565999999999 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041052+0000", 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7213000000000007e-05 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041163+0000", 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011066899999999999 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2534 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3604480~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8963382859999998, 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016482746999999999, 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:11.992 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2534 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024910+0000", 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1299999999999999e-07 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024912+0000", 2024-09-16T13:17:11.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024920+0000", 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024924+0000", 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.036e-06 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041197+0000", 2024-09-16T13:17:11.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016273242 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041282+0000", 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5013999999999996e-05 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041392+0000", 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109567 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:11.996 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:11.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:11.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:11.997 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2535 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:11.997 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:11.997 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8963176239999999, 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016691902000000002, 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2535 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7599999999999997e-07 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024932+0000", 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024949+0000", 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7053000000000001e-05 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024961+0000", 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:17:12.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041425+0000", 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016463851000000002 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041515+0000", 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9549999999999995e-05 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041622+0000", 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107132 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2536 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1638400~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.896279818, 2024-09-16T13:17:12.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016880869999999999, 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2536 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024971+0000", 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.615e-06 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024988+0000", 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6912999999999999e-05 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024993+0000", 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041653+0000", 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016660245000000001 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041743+0000", 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0099999999999995e-05 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041848+0000", 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105404 2024-09-16T13:17:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2537 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1671168~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8962490650000001, 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017075346000000002, 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2537 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024999+0000", 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025002+0000", 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:17:12.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025009+0000", 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025013+0000", 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1840000000000001e-06 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041886+0000", 2024-09-16T13:17:12.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016873249999999999 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041971+0000", 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4487000000000006e-05 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042074+0000", 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102847 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2538 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2621440~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8962288940000001, 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017314057000000001, 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2538 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.015 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025019+0000", 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025021+0000", 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025033+0000", 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1985e-05 2024-09-16T13:17:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025043+0000", 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3810000000000005e-06 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042107+0000", 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017064471000000001 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042203+0000", 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5810999999999995e-05 2024-09-16T13:17:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042333+0000", 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012956099999999999 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2539 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.896201166, 2024-09-16T13:17:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017500180000000001, 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2539 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025047+0000", 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025053+0000", 2024-09-16T13:17:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025058+0000", 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025061+0000", 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.072e-06 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042365+0000", 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017304885999999998 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042458+0000", 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2991000000000005e-05 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042546+0000", 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7904000000000004e-05 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.024 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.024 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:12.025 INFO:teuthology.orchestra.run.smithi098.stdout:1141 write 0xed615dd thru 0xed6e082 (0xcaa6 bytes) 2024-09-16T13:17:12.025 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:12.025 INFO:teuthology.orchestra.run.smithi098.stdout:1143 punch from 0xc37dc49 to 0xc38bd52, (0xe109 bytes) 2024-09-16T13:17:12.025 INFO:teuthology.orchestra.run.smithi098.stdout:1144 punch from 0x7a3f142 to 0x7a4723f, (0x80fd bytes) 2024-09-16T13:17:12.025 INFO:teuthology.orchestra.run.smithi098.stdout:1145 writesame 0x4f5b960 thru 0x4f634a2 data_size 0x1(0x7b43 bytes) 2024-09-16T13:17:12.026 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:12.026 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:12.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:12.028 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:12.030 INFO:teuthology.orchestra.run.smithi098.stdout:1146 punch from 0x8137d1c to 0x813cc7d, (0x4f61 bytes) 2024-09-16T13:17:12.030 INFO:teuthology.orchestra.run.smithi098.stdout:1147 read 0x72cc6cd thru 0x72d1ecd (0x5801 bytes) 2024-09-16T13:17:12.030 INFO:teuthology.orchestra.run.smithi098.stdout:1149 write 0xbc87b61 thru 0xbc91ad6 (0x9f76 bytes) 2024-09-16T13:17:12.030 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:12.030 INFO:teuthology.orchestra.run.smithi098.stdout:1151 write 0xc685b7b thru 0xc686edf (0x1365 bytes) 2024-09-16T13:17:12.030 INFO:teuthology.orchestra.run.smithi098.stdout:1154 write 0x2714e88 thru 0x2720645 (0xb7be bytes) 2024-09-16T13:17:12.030 INFO:teuthology.orchestra.run.smithi098.stdout:1155 read 0x8c61160 thru 0x8c62cff (0x1ba0 bytes) 2024-09-16T13:17:12.030 INFO:teuthology.orchestra.run.smithi098.stdout:1156 write 0xae80a9f thru 0xae81621 (0xb83 bytes) 2024-09-16T13:17:12.030 INFO:teuthology.orchestra.run.smithi098.stdout:1158 write 0x514a871 thru 0x514ed47 (0x44d7 bytes) 2024-09-16T13:17:12.031 INFO:teuthology.orchestra.run.smithi098.stdout:1159 write 0xcd8e93e thru 0xcd8f59b (0xc5e bytes) 2024-09-16T13:17:12.031 INFO:teuthology.orchestra.run.smithi098.stdout:1160 read 0xa2ebee0 thru 0xa2f84bb (0xc5dc bytes) 2024-09-16T13:17:12.031 INFO:teuthology.orchestra.run.smithi098.stdout:1163 read 0x9a1d2cb thru 0x9a25f52 (0x8c88 bytes) 2024-09-16T13:17:12.031 INFO:teuthology.orchestra.run.smithi098.stdout:1165 read 0x9e8aca thru 0x9f726d (0xe7a4 bytes) 2024-09-16T13:17:12.031 INFO:teuthology.orchestra.run.smithi098.stdout:1166 write 0x15dd94b thru 0x15de1f2 (0x8a8 bytes) 2024-09-16T13:17:12.031 INFO:teuthology.orchestra.run.smithi098.stdout:1167 write 0xb22be95 thru 0xb236a87 (0xabf3 bytes) 2024-09-16T13:17:12.031 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:12.020+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:12.051 INFO:teuthology.orchestra.run.smithi098.stdout:1168 trunc from 0xed6e083 to 0xfad27d 2024-09-16T13:17:12.085 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:17:12.110 INFO:teuthology.orchestra.run.smithi098.stdout:1169 writesame 0x8be778e thru 0x8bee518 data_size 0x1(0x6d8b bytes) 2024-09-16T13:17:12.126 INFO:teuthology.orchestra.run.smithi098.stdout:1170 write 0xd3c7f71 thru 0xd3cb6b9 (0x3749 bytes) 2024-09-16T13:17:12.126 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:12.122+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:12.135 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:12.148 INFO:teuthology.orchestra.run.smithi098.stdout:1172 writesame 0xe178cd8 thru 0xe186b6b data_size 0x37(0xde94 bytes) 2024-09-16T13:17:12.148 INFO:teuthology.orchestra.run.smithi098.stdout:1173 read 0xdfec8b4 thru 0xdff4df9 (0x8546 bytes) 2024-09-16T13:17:12.148 INFO:teuthology.orchestra.run.smithi098.stdout:1174 read 0xd13fca6 thru 0xd147095 (0x73f0 bytes) 2024-09-16T13:17:12.148 INFO:teuthology.orchestra.run.smithi098.stdout:1176 write 0x5491dd4 thru 0x549e1e8 (0xc415 bytes) 2024-09-16T13:17:12.148 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:12.148 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:12.149 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:12.149 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:12.156 INFO:teuthology.orchestra.run.smithi098.stdout:1177 punch from 0xdf507e1 to 0xdf5f05f, (0xe87e bytes) 2024-09-16T13:17:12.156 INFO:teuthology.orchestra.run.smithi098.stdout:1178 writesame 0x6fb6a43 thru 0x6fc19ae data_size 0x6d(0xaf6c bytes) 2024-09-16T13:17:12.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:12.169 INFO:teuthology.orchestra.run.smithi098.stdout:1179 write 0x592707c thru 0x59270c3 (0x48 bytes) 2024-09-16T13:17:12.175 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:12.192 INFO:teuthology.orchestra.run.smithi098.stdout:1182 trunc from 0xe186b6c to 0x629f47a 2024-09-16T13:17:12.213 INFO:teuthology.orchestra.run.smithi098.stdout:1184 read 0x6045ec8 thru 0x6049c3a (0x3d73 bytes) 2024-09-16T13:17:12.227 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:12.220+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:12.243 INFO:teuthology.orchestra.run.smithi098.stdout:1185 writesame 0xeccdfc0 thru 0xecd4e00 data_size 0x5(0x6e41 bytes) 2024-09-16T13:17:12.243 INFO:teuthology.orchestra.run.smithi098.stdout:1189 read 0xaa4f824 thru 0xaa5b995 (0xc172 bytes) 2024-09-16T13:17:12.243 INFO:teuthology.orchestra.run.smithi098.stdout:1193 write 0xd0ece72 thru 0xd0f6611 (0x97a0 bytes) 2024-09-16T13:17:12.248 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:12.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:12.248 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:12.248 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:12.248 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:1194 read 0x17b2c06 thru 0x17b4862 (0x1c5d bytes) 2024-09-16T13:17:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:1195 write 0x8d51206 thru 0x8d5bc95 (0xaa90 bytes) 2024-09-16T13:17:12.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:12.276 INFO:teuthology.orchestra.run.smithi098.stdout:1196 trunc from 0xecd4e01 to 0xa1e0ead 2024-09-16T13:17:12.297 INFO:teuthology.orchestra.run.smithi098.stdout:1197 punch from 0x2ee0b4e to 0x2ee668b, (0x5b3d bytes) 2024-09-16T13:17:12.297 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:12.297 INFO:teuthology.orchestra.run.smithi098.stdout:1199 read 0x5487b50 thru 0x548b8fc (0x3dad bytes) 2024-09-16T13:17:12.297 INFO:teuthology.orchestra.run.smithi098.stdout:1200 read 0x41fba08 thru 0x42080bc (0xc6b5 bytes) 2024-09-16T13:17:12.298 INFO:teuthology.orchestra.run.smithi098.stdout:1201 write 0x625f429 thru 0x626b164 (0xbd3c bytes) 2024-09-16T13:17:12.308 INFO:teuthology.orchestra.run.smithi098.stdout:1203 punch from 0x418bdbd to 0x4197a49, (0xbc8c bytes) 2024-09-16T13:17:12.318 INFO:teuthology.orchestra.run.smithi098.stdout:1204 trunc from 0xa1e0ead to 0xe2f1ec4 2024-09-16T13:17:12.328 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:12.320+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:12.330 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:12.330 INFO:teuthology.orchestra.run.smithi098.stdout:1206 read 0x7b0d6cc thru 0x7b0f7fa (0x212f bytes) 2024-09-16T13:17:12.330 INFO:teuthology.orchestra.run.smithi098.stdout:1208 writesame 0xd2ea982 thru 0xd2f0681 data_size 0x7c(0x5d00 bytes) 2024-09-16T13:17:12.337 INFO:teuthology.orchestra.run.smithi098.stdout:1212 writesame 0x407de0 thru 0x40ff58 data_size 0x7(0x8179 bytes) 2024-09-16T13:17:12.349 INFO:teuthology.orchestra.run.smithi098.stdout:1213 write 0x443008 thru 0x443043 (0x3c bytes) 2024-09-16T13:17:12.355 INFO:teuthology.orchestra.run.smithi098.stdout:1214 write 0xc357e05 thru 0xc36583d (0xda39 bytes) 2024-09-16T13:17:12.359 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1022 3.10 3:0c4f3e94:::rbd_data.112de98401a1.000000000000002d:head [set-alloc-hint object_size 4194304 write_size 4194304,write 830679~9355] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:58.732415+0000", 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.619069570000001, 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040114139999999996, 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.360 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1022 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732415+0000", 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732415+0000", 2024-09-16T13:17:12.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732418+0000", 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2770000000000001e-06 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732942+0000", 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052370100000000001 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.732986+0000", 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4079000000000003e-05 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733002+0000", 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5716999999999999e-05 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733137+0000", 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013541700000000001 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733389+0000", 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025132399999999998 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.733884+0000", 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049490199999999997 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736097+0000", 2024-09-16T13:17:12.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002213338 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736272+0000", 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017514900000000001 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736363+0000", 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1467999999999999e-05 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:58.736427+0000", 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3041999999999997e-05 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1482 3.a 3:535f7855:::rbd_data.112de98401a1.0000000000000012:head [truncate 2391135] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:00.072537+0000", 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.278948088, 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069626239999999997, 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.366 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1482 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072537+0000", 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072537+0000", 2024-09-16T13:17:12.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072539+0000", 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7829999999999999e-06 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072551+0000", 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1249e-05 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072588+0000", 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7230000000000001e-05 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072601+0000", 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3512999999999999e-05 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072737+0000", 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000135387 2024-09-16T13:17:12.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.072747+0000", 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0645999999999999e-05 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.076771+0000", 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040232200000000001 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.076893+0000", 2024-09-16T13:17:12.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122731 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.077095+0000", 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020168900000000001 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.078550+0000", 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001454711 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.079360+0000", 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00081076499999999997 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.079438+0000", 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7879000000000004e-05 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:00.079499+0000", 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820999999999999e-05 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.372 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1835 3.a 3:535385f0:::rbd_children:head [call rbd.add_child in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:02.969751+0000", 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.3817337970000008, 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067434870000000003, 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1835 2024-09-16T13:17:12.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969751+0000", 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969751+0000", 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969753+0000", 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4049999999999998e-06 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969763+0000", 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0067e-05 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969790+0000", 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6842000000000002e-05 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969799+0000", 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8789999999999992e-06 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969867+0000", 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8363999999999998e-05 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.969874+0000", 2024-09-16T13:17:12.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4119999999999998e-06 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.972618+0000", 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027437809999999998 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.972799+0000", 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018139799999999999 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.973042+0000", 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024340899999999999 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.976365+0000", 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033225809999999998 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.976390+0000", 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5364e-05 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.976452+0000", 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1464000000000006e-05 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.976494+0000", 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2521000000000002e-05 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.379 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1868 3.10 3:0bfff005:::rbd_data.112d2ce070e7.0000000000000019:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:02.999819+0000", 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.3516654389999996, 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042038400000000004, 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1868 2024-09-16T13:17:12.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999819+0000", 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999819+0000", 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999821+0000", 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999830+0000", 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3350000000000007e-06 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999853+0000", 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3445999999999999e-05 2024-09-16T13:17:12.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999863+0000", 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0278000000000001e-05 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:02.999955+0000", 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1396000000000004e-05 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.000109+0000", 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015418499999999999 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.000429+0000", 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032062300000000002 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.003908+0000", 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034785160000000001 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.003941+0000", 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2546999999999999e-05 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.003986+0000", 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4990999999999997e-05 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.004023+0000", 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7531999999999999e-05 2024-09-16T13:17:12.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1874 3.a 3:55586239:::rbd_data.112d2ce070e7.000000000000001d:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.002278+0000", 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.3492067910000003, 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048428020000000002, 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.386 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1874 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002278+0000", 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002278+0000", 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002279+0000", 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2529999999999999e-06 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002285+0000", 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7080000000000002e-06 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002303+0000", 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8026999999999999e-05 2024-09-16T13:17:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002310+0000", 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7750000000000004e-06 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002374+0000", 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4634000000000005e-05 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.002534+0000", 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016008999999999999 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.007121+0000", 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045863149999999997 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1886 3.a 3:55586239:::rbd_data.112d2ce070e7.000000000000001d:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.008537+0000", 2024-09-16T13:17:12.390 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.3429480379999994, 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006065801, 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1886 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008537+0000", 2024-09-16T13:17:12.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008537+0000", 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008538+0000", 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.128e-06 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008544+0000", 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2650000000000002e-06 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008556+0000", 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2132e-05 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008562+0000", 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1129999999999997e-06 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008653+0000", 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1044000000000004e-05 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008746+0000", 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2497999999999994e-05 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.008930+0000", 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018463900000000001 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012488+0000", 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035579790000000002 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014501+0000", 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020121340000000001 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014559+0000", 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7967000000000003e-05 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014602+0000", 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3902000000000001e-05 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:1894 3.1e e31/23 3:7c25ca39:::rbd_data.112d2ce070e7.0000000000000024:head v 31'2, mlcod=0'0)", 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.010400+0000", 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.3410846660000004, 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041542200000000001, 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.396 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010400+0000", 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010400+0000", 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010401+0000", 2024-09-16T13:17:12.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6899999999999997e-07 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010406+0000", 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8289999999999996e-06 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010416+0000", 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6679999999999996e-06 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010420+0000", 2024-09-16T13:17:12.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010493+0000", 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3345999999999999e-05 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.010525+0000", 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1677000000000002e-05 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014522+0000", 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039971540000000002 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014554+0000", 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1906000000000003e-05 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1897 3.10 3:0b516a8b:::rbd_data.112d2ce070e7.0000000000000025:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.012651+0000", 2024-09-16T13:17:12.400 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.3388334650000004, 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046366310000000004, 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1897 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012651+0000", 2024-09-16T13:17:12.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012651+0000", 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012652+0000", 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.102e-06 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012659+0000", 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8580000000000002e-06 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012677+0000", 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7880000000000002e-05 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012684+0000", 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4159999999999998e-06 2024-09-16T13:17:12.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012729+0000", 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4883999999999999e-05 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.012815+0000", 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6019000000000004e-05 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.013058+0000", 2024-09-16T13:17:12.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024225399999999999 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.014458+0000", 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001400016 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.017180+0000", 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027226160000000002 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.017232+0000", 2024-09-16T13:17:12.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1452000000000001e-05 2024-09-16T13:17:12.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.017288+0000", 2024-09-16T13:17:12.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6134000000000002e-05 2024-09-16T13:17:12.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.409 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:1925 3.a 3:574306b8:::rbd_data.112d2ce070e7.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,truncate 1179648] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.042069+0000", 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.3094161199999999, 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045267179999999999, 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.410 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1925 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042069+0000", 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042069+0000", 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042071+0000", 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8200000000000001e-06 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042081+0000", 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8269999999999993e-06 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042106+0000", 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4858000000000002e-05 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042115+0000", 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9409999999999996e-06 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042182+0000", 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6631000000000006e-05 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042307+0000", 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012550699999999999 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.042451+0000", 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014376300000000001 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.043788+0000", 2024-09-16T13:17:12.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001337095 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.046474+0000", 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026856419999999998 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.046546+0000", 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2175000000000006e-05 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.046595+0000", 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9459000000000002e-05 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:5174 3.a 3:548e7ca7:::rbd_data.112d64676e7f.00000000000002f3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:07.863709+0000", 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.4877760750000002, 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010653861000000001, 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.416 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 5174 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863709+0000", 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863709+0000", 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863711+0000", 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1399999999999998e-06 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863721+0000", 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0727000000000001e-05 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863757+0000", 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5525999999999998e-05 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863770+0000", 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2595999999999999e-05 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.863897+0000", 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012721100000000001 2024-09-16T13:17:12.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.864134+0000", 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000237144 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.864431+0000", 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029679400000000002 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.866015+0000", 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001584366 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.874278+0000", 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0082627359999999997 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.874316+0000", 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8427999999999999e-05 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:07.874362+0000", 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6193000000000003e-05 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:6726 3.a 3:50008994:::rbd_object_map.112d6fb8d387:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:11.976178+0000", 2024-09-16T13:17:12.422 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.37530659, 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037979860000000002, 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6726 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.976178+0000", 2024-09-16T13:17:12.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.976178+0000", 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.976181+0000", 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.067e-06 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.976195+0000", 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4066999999999999e-05 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.976229+0000", 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4199000000000002e-05 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.976247+0000", 2024-09-16T13:17:12.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7493e-05 2024-09-16T13:17:12.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.976437+0000", 2024-09-16T13:17:12.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019008499999999999 2024-09-16T13:17:12.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.976746+0000", 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030935400000000002 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.977551+0000", 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080498500000000003 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.978901+0000", 2024-09-16T13:17:12.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001349797 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.979817+0000", 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00091629099999999996 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.979891+0000", 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4086999999999997e-05 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.979976+0000", 2024-09-16T13:17:12.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4561000000000006e-05 2024-09-16T13:17:12.429 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.429 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.429 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.429 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:6728 3.a 3:50008994:::rbd_object_map.112d6fb8d387:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T13:17:12.429 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:11.982938+0000", 2024-09-16T13:17:12.429 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.36854629, 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042363390000000004, 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6728 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.982938+0000", 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.982938+0000", 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.982940+0000", 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.108e-06 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.982950+0000", 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9860000000000006e-06 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.982977+0000", 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7008e-05 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.982988+0000", 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0794e-05 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.983086+0000", 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7371e-05 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.983221+0000", 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013511 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.983658+0000", 2024-09-16T13:17:12.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00043742800000000001 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.986850+0000", 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003191419 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.986934+0000", 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4374000000000002e-05 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.987066+0000", 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013253000000000001 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.987175+0000", 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108211 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.436 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:6729 3.a 3:50008994:::rbd_object_map.112d6fb8d387:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T13:17:12.436 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:11.983011+0000", 2024-09-16T13:17:12.436 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.36847399800000002, 2024-09-16T13:17:12.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043344270000000001, 2024-09-16T13:17:12.436 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.436 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.436 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.436 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.436 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6729 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.983011+0000", 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.983011+0000", 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.983012+0000", 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7600000000000001e-06 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.983021+0000", 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2600000000000005e-06 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.983032+0000", 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1285e-05 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.983061+0000", 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8588000000000001e-05 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.984347+0000", 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012868280000000001 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.984534+0000", 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018686399999999999 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.985685+0000", 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001150765 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.987232+0000", 2024-09-16T13:17:12.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001546735 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.987275+0000", 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3597999999999998e-05 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.987315+0000", 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9301999999999997e-05 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:11.987345+0000", 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0442000000000001e-05 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:6743 3.1c 3:39ffe635:::rbd_data.112d6fb8d387.0000000000000055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:12.057977+0000", 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.29350754800000001, 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047861830000000003, 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6743 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.057977+0000", 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.057977+0000", 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.057979+0000", 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.04e-06 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.057989+0000", 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.013e-05 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058018+0000", 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8526000000000001e-05 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058031+0000", 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.328e-05 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058193+0000", 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016153000000000001 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058456+0000", 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026374599999999998 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058706+0000", 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024948 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.060780+0000", 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020739970000000002 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.062675+0000", 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001895435 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.062725+0000", 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9753999999999998e-05 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.062763+0000", 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8265e-05 2024-09-16T13:17:12.447 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:6745 3.a 3:51f0ceb7:::rbd_data.112d6fb8d387.0000000000000072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:12.058087+0000", 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.293397976, 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054755050000000003, 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6745 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058087+0000", 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058087+0000", 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058088+0000", 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4079999999999999e-06 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058094+0000", 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9839999999999998e-06 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058113+0000", 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9344999999999999e-05 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058123+0000", 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3419999999999997e-06 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058216+0000", 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3513999999999999e-05 2024-09-16T13:17:12.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058365+0000", 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149223 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.058571+0000", 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020599099999999999 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.060806+0000", 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022340839999999999 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.063417+0000", 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026109620000000001 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.063513+0000", 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6952000000000003e-05 2024-09-16T13:17:12.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.063562+0000", 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8699999999999998e-05 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:6765 3.10 3:0df2f605:::rbd_data.112d6fb8d387.00000000000002df:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:12.066583+0000", 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.28490116199999999, 2024-09-16T13:17:12.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003886521, 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6765 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.066583+0000", 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.066583+0000", 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.066585+0000", 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3039999999999999e-06 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.066591+0000", 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5130000000000002e-06 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.066613+0000", 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1195999999999998e-05 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.066621+0000", 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4039999999999995e-06 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.066678+0000", 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6660999999999998e-05 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.066759+0000", 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1865000000000003e-05 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.066877+0000", 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117559 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.067572+0000", 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00069534900000000005 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.070391+0000", 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028182200000000002 2024-09-16T13:17:12.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.070442+0000", 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1141999999999999e-05 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.070470+0000", 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8308000000000001e-05 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:6816 3.a 3:50008994:::rbd_object_map.112d6fb8d387:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:12.153363+0000", 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.198121305, 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0052533349999999996, 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6816 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.461 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153363+0000", 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153363+0000", 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153366+0000", 2024-09-16T13:17:12.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5440000000000001e-06 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153380+0000", 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3766e-05 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153419+0000", 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9048999999999998e-05 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153435+0000", 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6575999999999999e-05 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153552+0000", 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117049 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153780+0000", 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022781799999999999 2024-09-16T13:17:12.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.154727+0000", 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00094730499999999996 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.158344+0000", 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036165220000000001 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.158452+0000", 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010828 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.158536+0000", 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3712000000000001e-05 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.158617+0000", 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0714e-05 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.466 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:6817 3.a 3:50008994:::rbd_object_map.112d6fb8d387:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:12.153480+0000", 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.19800490100000001, 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056156260000000003, 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6817 2024-09-16T13:17:12.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153480+0000", 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153480+0000", 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153481+0000", 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.097e-06 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153489+0000", 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3100000000000001e-06 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153520+0000", 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0855999999999997e-05 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.153529+0000", 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8259999999999999e-06 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.155841+0000", 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002312279 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.156065+0000", 2024-09-16T13:17:12.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022432499999999999 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.157169+0000", 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011031319999999999 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.158789+0000", 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016207210000000001 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.158929+0000", 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013940600000000001 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.159022+0000", 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3205e-05 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.159095+0000", 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3468999999999997e-05 2024-09-16T13:17:12.472 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:6862 3.a 3:50008994:::rbd_object_map.112d6fb8d387:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:12.295082+0000", 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.056402610999999998, 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051190469999999998, 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.473 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6862 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295082+0000", 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295082+0000", 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295083+0000", 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.232e-06 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295090+0000", 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2219999999999996e-06 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295108+0000", 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7980000000000001e-05 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295116+0000", 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.182e-06 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295235+0000", 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011956900000000001 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295450+0000", 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021494099999999999 2024-09-16T13:17:12.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.296391+0000", 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00094123600000000003 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.299853+0000", 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003461703 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.299956+0000", 2024-09-16T13:17:12.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010262799999999999 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.300115+0000", 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015967299999999999 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.300201+0000", 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5680999999999995e-05 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:6863 3.a 3:50008994:::rbd_object_map.112d6fb8d387:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:12.295140+0000", 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.056345106999999998, 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063736440000000004, 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6863 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295140+0000", 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295140+0000", 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295140+0000", 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8800000000000002e-07 2024-09-16T13:17:12.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295144+0000", 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9400000000000004e-06 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295153+0000", 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3869999999999995e-06 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.295157+0000", 2024-09-16T13:17:12.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6789999999999996e-06 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.297494+0000", 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023367190000000001 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.297753+0000", 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025856099999999998 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:12.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.298588+0000", 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00083488500000000005 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.301077+0000", 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024895609999999999 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.301383+0000", 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030601000000000001 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.301462+0000", 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8721999999999996e-05 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:12.301513+0000", 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1391999999999999e-05 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.484 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:12.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:12.488 INFO:teuthology.orchestra.run.smithi098.stdout:1216 write 0x7ef83af thru 0x7f071a5 (0xedf7 bytes) 2024-09-16T13:17:12.488 INFO:teuthology.orchestra.run.smithi098.stdout:1217 write 0xedeeccd thru 0xedfca10 (0xdd44 bytes) 2024-09-16T13:17:12.488 INFO:teuthology.orchestra.run.smithi098.stdout:1218 write 0xe3d0a6e thru 0xe3d3127 (0x26ba bytes) 2024-09-16T13:17:12.488 INFO:teuthology.orchestra.run.smithi098.stdout:1220 read 0x5d6e55e thru 0x5d7017d (0x1c20 bytes) 2024-09-16T13:17:12.488 INFO:teuthology.orchestra.run.smithi098.stdout:1221 read 0x4e11a47 thru 0x4e1c308 (0xa8c2 bytes) 2024-09-16T13:17:12.488 INFO:teuthology.orchestra.run.smithi098.stdout:1222 trunc from 0xedfca11 to 0xbd237d5 2024-09-16T13:17:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:1223 read 0x12686d2 thru 0x1269051 (0x980 bytes) 2024-09-16T13:17:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:1228 read 0x70796d5 thru 0x707f224 (0x5b50 bytes) 2024-09-16T13:17:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:1229 write 0xc141b26 thru 0xc1483d5 (0x68b0 bytes) 2024-09-16T13:17:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:1230 write 0x4cbc43a thru 0x4cbd11d (0xce4 bytes) 2024-09-16T13:17:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:1232 read 0x866ec1 thru 0x86c147 (0x5287 bytes) 2024-09-16T13:17:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:1233 punch from 0x22d4b21 to 0x22db1fd, (0x66dc bytes) 2024-09-16T13:17:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:1235 write 0x27bb4d4 thru 0x27c0de7 (0x5914 bytes) 2024-09-16T13:17:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:1236 writesame 0x7915b2e thru 0x7922f61 data_size 0x1b(0xd434 bytes) 2024-09-16T13:17:12.489 INFO:teuthology.orchestra.run.smithi098.stdout:1238 punch from 0x75fb4f5 to 0x76071a0, (0xbcab bytes) 2024-09-16T13:17:12.490 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:12.480+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:12.490 INFO:teuthology.orchestra.run.smithi098.stdout:1239 writesame 0xe33f847 thru 0xe349fd5 data_size 0x17(0xa78f bytes) 2024-09-16T13:17:12.490 INFO:teuthology.orchestra.run.smithi098.stdout:1240 writesame 0xd1e628b thru 0xd1f5b50 data_size 0x7(0xf8c6 bytes) 2024-09-16T13:17:12.506 INFO:teuthology.orchestra.run.smithi098.stdout:1242 write 0x26a3695 thru 0x26a4a7e (0x13ea bytes) 2024-09-16T13:17:12.513 INFO:teuthology.orchestra.run.smithi098.stdout:1243 write 0xb89b5bc thru 0xb8a072f (0x5174 bytes) 2024-09-16T13:17:12.522 INFO:teuthology.orchestra.run.smithi098.stdout:1244 write 0x3b9609e thru 0x3b9c9a7 (0x690a bytes) 2024-09-16T13:17:12.552 INFO:teuthology.orchestra.run.smithi098.stdout:1245 trunc from 0xe349fd6 to 0x1cf73fc 2024-09-16T13:17:12.587 INFO:teuthology.orchestra.run.smithi098.stdout:1246 trunc from 0x1cf73fc to 0x75cbf46 2024-09-16T13:17:12.587 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:12.580+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:12.597 INFO:teuthology.orchestra.run.smithi098.stdout:1247 read 0x312765 thru 0x31f45d (0xccf9 bytes) 2024-09-16T13:17:12.598 INFO:teuthology.orchestra.run.smithi098.stdout:1248 punch from 0x58a6e6d to 0x58a8071, (0x1204 bytes) 2024-09-16T13:17:12.598 INFO:teuthology.orchestra.run.smithi098.stdout:1250 write 0x6e4b2b1 thru 0x6e5811a (0xce6a bytes) 2024-09-16T13:17:12.601 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:12.601 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:12.602 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:12.602 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:12.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:12.627 INFO:teuthology.orchestra.run.smithi098.stdout:1252 writesame 0xccfe9b3 thru 0xccffcc6 data_size 0xc(0x1314 bytes) 2024-09-16T13:17:12.628 INFO:teuthology.orchestra.run.smithi098.stdout:1254 write 0x124233e thru 0x1248501 (0x61c4 bytes) 2024-09-16T13:17:12.636 INFO:teuthology.orchestra.run.smithi098.stdout:1255 trunc from 0xccffcc7 to 0xd8c0937 2024-09-16T13:17:12.647 INFO:teuthology.orchestra.run.smithi098.stdout:1256 write 0x847af36 thru 0x84817ca (0x6895 bytes) 2024-09-16T13:17:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:1258 read 0x227d5bc thru 0x22816a4 (0x40e9 bytes) 2024-09-16T13:17:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:1259 write 0xbc5d19b thru 0xbc66f20 (0x9d86 bytes) 2024-09-16T13:17:12.665 INFO:teuthology.orchestra.run.smithi098.stdout:1260 write 0x6234dcb thru 0x6237edd (0x3113 bytes) 2024-09-16T13:17:12.671 INFO:teuthology.orchestra.run.smithi098.stdout:1261 write 0xbe8dbd7 thru 0xbe9be07 (0xe231 bytes) 2024-09-16T13:17:12.677 INFO:teuthology.orchestra.run.smithi098.stdout:1262 punch from 0x89e5745 to 0x89f5689, (0xff44 bytes) 2024-09-16T13:17:12.677 INFO:teuthology.orchestra.run.smithi098.stdout:1263 punch from 0x7caad4d to 0x7cb0afd, (0x5db0 bytes) 2024-09-16T13:17:12.688 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:12.682+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:12.691 INFO:teuthology.orchestra.run.smithi098.stdout:1264 trunc from 0xd8c0937 to 0x7ce9c20 2024-09-16T13:17:12.715 INFO:teuthology.orchestra.run.smithi098.stdout:1265 write 0x644645 thru 0x64e8d0 (0xa28c bytes) 2024-09-16T13:17:12.717 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:12.717 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:12.717 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:12.717 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:12.718 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:12.721 INFO:teuthology.orchestra.run.smithi098.stdout:1266 clone 6 order 19 su 4096 sc 14 2024-09-16T13:17:12.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:12.789 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:12.782+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:12.831 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.800324525000001, 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:17:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:17:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.800177118999997, 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:17:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.800149775999998, 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:17:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.241474513999997, 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:17:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:17:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.241386921999997, 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:17:12.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:17:12.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.241216999999999, 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:17:12.861 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:17:12.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:17:12.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2259 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3014656~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0898884239999997, 2024-09-16T13:17:12.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016650893999999999, 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2259 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:12.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735831+0000", 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9530000000000005e-06 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735836+0000", 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7450000000000003e-06 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735839+0000", 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.915e-06 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752296+0000", 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016457241000000001 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752360+0000", 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4108000000000004e-05 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752473+0000", 2024-09-16T13:17:12.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112305 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2260 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0898685029999999, 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016799073000000001, 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.872 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2260 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:12.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735843+0000", 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3480000000000001e-06 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735847+0000", 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5990000000000002e-06 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735849+0000", 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5859999999999999e-06 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752509+0000", 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016659377 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752559+0000", 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9774000000000002e-05 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752641+0000", 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2057000000000004e-05 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2261 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0898563140000004, 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016960698, 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2261 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.03e-07 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735855+0000", 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735860+0000", 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1110000000000001e-06 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735862+0000", 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:12.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752695+0000", 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01683343 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752744+0000", 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8414000000000001e-05 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752814+0000", 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0623000000000005e-05 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2262 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1048576~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0898456999999997, 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017224107999999998, 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.881 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2262 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735865+0000", 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.91e-07 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735866+0000", 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1480000000000001e-06 2024-09-16T13:17:12.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735870+0000", 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735877+0000", 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4370000000000001e-06 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752871+0000", 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016994061000000001 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752962+0000", 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0470000000000006e-05 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753089+0000", 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126642 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.885 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2263 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0898302409999996, 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017392088999999999, 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2263 2024-09-16T13:17:12.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-07 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735881+0000", 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735885+0000", 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4180000000000001e-06 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735887+0000", 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-06 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753116+0000", 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017228963 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753184+0000", 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7374999999999998e-05 2024-09-16T13:17:12.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753272+0000", 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8323000000000001e-05 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2264 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2064384~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:12.890 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0898203899999999, 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017630541999999999, 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2264 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735891+0000", 2024-09-16T13:17:12.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735895+0000", 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735897+0000", 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4669999999999998e-06 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753316+0000", 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017418685 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753403+0000", 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7590000000000007e-05 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753520+0000", 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116863 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2265 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1081344~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0898105079999993, 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017739263000000002, 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2265 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735901+0000", 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735904+0000", 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.304e-06 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735907+0000", 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3599999999999999e-06 2024-09-16T13:17:12.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753534+0000", 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017627446000000001 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753576+0000", 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1474999999999998e-05 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753639+0000", 2024-09-16T13:17:12.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3188999999999994e-05 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2266 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2097152~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0898012749999992, 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017970893000000002, 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2266 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735914+0000", 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1699999999999996e-06 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735918+0000", 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6859999999999999e-06 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735920+0000", 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4870000000000001e-06 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753685+0000", 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017764582000000001 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753771+0000", 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6259000000000002e-05 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753880+0000", 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010844300000000001 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2267 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:12.903 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0897873740000001, 2024-09-16T13:17:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018110804000000001, 2024-09-16T13:17:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2267 2024-09-16T13:17:12.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735924+0000", 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735928+0000", 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.416e-06 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735930+0000", 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753898+0000", 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017967852999999999 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753955+0000", 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6662000000000001e-05 2024-09-16T13:17:12.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754034+0000", 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8904000000000002e-05 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2268 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3080192~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0897778890000005, 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018252866, 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.908 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2268 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735934+0000", 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0950000000000001e-06 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735938+0000", 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735941+0000", 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8750000000000002e-06 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754071+0000", 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018129941 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754120+0000", 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8359000000000002e-05 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754185+0000", 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5358999999999995e-05 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2269 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0897665100000005, 2024-09-16T13:17:12.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018533516, 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2269 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735945+0000", 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1149999999999999e-06 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735948+0000", 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5250000000000001e-06 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735951+0000", 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9909999999999998e-06 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754270+0000", 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018318081999999999 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754325+0000", 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5648e-05 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754477+0000", 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015188199999999999 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.916 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2270 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1114112~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0897560409999993, 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019103282999999999, 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2270 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735955+0000", 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-16T13:17:12.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735956+0000", 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.096e-06 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735959+0000", 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5140000000000002e-06 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735963+0000", 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.053e-06 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754822+0000", 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018858883 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754941+0000", 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118789 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755057+0000", 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116272 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2271 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1146880~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0897342289999994, 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019332309999999998, 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.921 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2271 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:12.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3500000000000002e-07 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735982+0000", 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735987+0000", 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1930000000000003e-06 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735993+0000", 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4659999999999996e-06 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755081+0000", 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019088187 2024-09-16T13:17:12.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755194+0000", 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011223299999999999 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755308+0000", 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114663 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2272 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 163840~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.0897123989999997, 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019476291, 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2272 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:12.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736003+0000", 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2279999999999998e-06 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736007+0000", 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.506e-06 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736011+0000", 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9280000000000003e-06 2024-09-16T13:17:12.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755336+0000", 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019325189999999999 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755393+0000", 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7293999999999997e-05 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755474+0000", 2024-09-16T13:17:12.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0635000000000003e-05 2024-09-16T13:17:12.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.930 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:12.930 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:12.930 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:12.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:12.932 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:12.924+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:13.032 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:13.026+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:13.039 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:13.040 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:13.040 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:13.040 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:13.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:13.133 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:13.126+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:13.152 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:13.152 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:13.152 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:13.152 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:13.152 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:13.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:13.208 INFO:teuthology.orchestra.run.smithi098.stdout:flattening image image_client.0-clone5 2024-09-16T13:17:13.233 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:13.226+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:13.272 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:51 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'33, mlcod=16'27)", 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.226058363999996, 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065364949999999998, 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038562+0000", 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0920000000000001e-06 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038571+0000", 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1339999999999993e-06 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038600+0000", 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9254000000000001e-05 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038609+0000", 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6049999999999998e-06 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.038982+0000", 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037359699999999998 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039013+0000", 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0945000000000003e-05 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045052+0000", 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060389490000000001 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045097+0000", 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4919000000000001e-05 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:52 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'34, mlcod=16'29)", 2024-09-16T13:17:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.039817+0000", 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.224802517999997, 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057720159999999996, 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039817+0000", 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039817+0000", 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039818+0000", 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.223e-06 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039828+0000", 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0125e-05 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039851+0000", 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3485999999999998e-05 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039861+0000", 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909999999999996e-06 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039933+0000", 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1829999999999995e-05 2024-09-16T13:17:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.039967+0000", 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4375999999999997e-05 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045505+0000", 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055381470000000002 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.045589+0000", 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3537999999999999e-05 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:55 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'37, mlcod=16'33)", 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.216486580000002, 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062294560000000004, 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048133+0000", 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5000000000000002e-07 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048240+0000", 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107209 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048252+0000", 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1263999999999999e-05 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048263+0000", 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1685000000000001e-05 2024-09-16T13:17:13.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048355+0000", 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2033999999999993e-05 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048378+0000", 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3062e-05 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054305+0000", 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059262639999999997 2024-09-16T13:17:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054362+0000", 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7287999999999998e-05 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:56 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'38, mlcod=16'33)", 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.215770562000003, 2024-09-16T13:17:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055458679999999998, 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:17:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048849+0000", 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0800000000000004e-07 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048942+0000", 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2954000000000005e-05 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048954+0000", 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2234e-05 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.048961+0000", 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0530000000000003e-06 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.049066+0000", 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105079 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.049088+0000", 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2427999999999999e-05 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054366+0000", 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052782189999999998 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.054394+0000", 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8192999999999999e-05 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4207.0:80 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'58, mlcod=16'53)", 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:35.604358+0000", 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 37.660260923000003, 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0060398550000000002, 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604358+0000", 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604358+0000", 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604359+0000", 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.054e-06 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604451+0000", 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1916000000000006e-05 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604469+0000", 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7376000000000001e-05 2024-09-16T13:17:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.604477+0000", 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0870000000000007e-06 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605124+0000", 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00064696399999999998 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.605155+0000", 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1259e-05 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.610343+0000", 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051882860000000003 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:35.610398+0000", 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4913000000000001e-05 2024-09-16T13:17:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1266 3.1 3:8739a53c:::rbd_data.112de98401a1.000000000000000b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.500416+0000", 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.764203431, 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055098930000000001, 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1266 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500416+0000", 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500416+0000", 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500417+0000", 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1829999999999999e-06 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500425+0000", 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9880000000000004e-06 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500441+0000", 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5614e-05 2024-09-16T13:17:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500451+0000", 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0071000000000001e-05 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500577+0000", 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012604700000000001 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.500769+0000", 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019282699999999999 2024-09-16T13:17:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501049+0000", 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027978600000000002 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503155+0000", 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021059630000000002 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.505807+0000", 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026514199999999998 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.505880+0000", 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2960999999999995e-05 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.505926+0000", 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6032999999999998e-05 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:1858 3.17 3:efe05268:::rbd_data.112d2ce070e7.0000000000000014:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:02.992700+0000", 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.271919069000001, 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0059941489999999998, 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 1858 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992700+0000", 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992700+0000", 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992703+0000", 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3859999999999999e-06 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992714+0000", 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0396e-05 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992747+0000", 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2753000000000003e-05 2024-09-16T13:17:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992776+0000", 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9159e-05 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.992869+0000", 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3700000000000001e-05 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.993141+0000", 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027109399999999999 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:02.998694+0000", 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055536609999999997 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2445 3.6 3:62f82d6c:::rbd_data.112dbfd5d823.0000000000000153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.010787+0000", 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.2538316760000008, 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049181609999999999, 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2445 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.010787+0000", 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.010787+0000", 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.010789+0000", 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.279e-06 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.010795+0000", 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3980000000000004e-06 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.010817+0000", 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1489999999999999e-05 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.010824+0000", 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2949999999999996e-06 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.010894+0000", 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9616000000000006e-05 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011050+0000", 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000155993 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011236+0000", 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018685699999999999 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014824+0000", 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035872410000000001 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015616+0000", 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00079192300000000004 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015666+0000", 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0309999999999998e-05 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015706+0000", 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9759000000000003e-05 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2457 3.1 3:81814fad:::rbd_data.112dbfd5d823.000000000000029d:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.016061+0000", 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.2485582659999999, 2024-09-16T13:17:13.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050659750000000003, 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2457 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016061+0000", 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016061+0000", 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016062+0000", 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9600000000000008e-07 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016067+0000", 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0749999999999997e-06 2024-09-16T13:17:13.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016083+0000", 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5829999999999999e-05 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016089+0000", 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1759999999999998e-06 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016137+0000", 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7556999999999999e-05 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016216+0000", 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9456999999999995e-05 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016305+0000", 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9160999999999996e-05 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.017392+0000", 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001086563 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.021013+0000", 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036216149999999999 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.021077+0000", 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3236000000000002e-05 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.021127+0000", 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0309000000000002e-05 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2458 3.12 3:4cc92853:::rbd_data.112dbfd5d823.000000000000029e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.016102+0000", 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.2485174449999992, 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050191230000000003, 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2458 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016102+0000", 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016102+0000", 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016102+0000", 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016104+0000", 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.984e-06 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016111+0000", 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9029999999999998e-06 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016114+0000", 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3340000000000002e-06 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016197+0000", 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2478999999999994e-05 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016265+0000", 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7595000000000006e-05 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.016341+0000", 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6932000000000001e-05 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.017440+0000", 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010985439999999999 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.021026+0000", 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035859849999999999 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.021086+0000", 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0242999999999999e-05 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.021121+0000", 2024-09-16T13:17:13.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4613999999999998e-05 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:4313 3.17 3:ef343474:::rbd_object_map.112de3ebec3a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.129919+0000", 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.1346999389999999, 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004977085, 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4313 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.129919+0000", 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.129919+0000", 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.129922+0000", 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3539999999999998e-06 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.129932+0000", 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.049e-05 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.129954+0000", 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2323999999999998e-05 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.129974+0000", 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9831e-05 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130036+0000", 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1958999999999994e-05 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130102+0000", 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5964999999999996e-05 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130490+0000", 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00038788499999999998 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.133027+0000", 2024-09-16T13:17:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002537157 2024-09-16T13:17:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.134786+0000", 2024-09-16T13:17:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017586419999999999 2024-09-16T13:17:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.134848+0000", 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2310000000000005e-05 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.134896+0000", 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8167999999999997e-05 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:4315 3.17 3:ef343474:::rbd_object_map.112de3ebec3a:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:13.330 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.130339+0000", 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.1342798690000002, 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046623519999999998, 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 4315 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130339+0000", 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130339+0000", 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130340+0000", 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1260000000000001e-06 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130347+0000", 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0090000000000001e-06 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130362+0000", 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4422e-05 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130372+0000", 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8160000000000005e-06 2024-09-16T13:17:13.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130882+0000", 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00051026800000000003 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.130993+0000", 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000111409 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 5", 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.131345+0000", 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035211699999999999 2024-09-16T13:17:13.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.133034+0000", 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016888649999999999 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.134948+0000", 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00191325 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.134978+0000", 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.048e-05 2024-09-16T13:17:13.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.135002+0000", 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3589999999999999e-05 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:4378 3.b e34/23 3:d604d247:::rbd_header.112dbfd5d823:head v 34'83, mlcod=32'81)", 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.351967+0000", 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.9126522330000002, 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0073312330000000004, 2024-09-16T13:17:13.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.351967+0000", 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.351967+0000", 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.351968+0000", 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4440000000000001e-06 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.351979+0000", 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0453e-05 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.352002+0000", 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2799999999999999e-05 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.352010+0000", 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5089999999999993e-06 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.352073+0000", 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2494000000000002e-05 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.352110+0000", 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7580999999999998e-05 2024-09-16T13:17:13.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.359233+0000", 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071225790000000004 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.359298+0000", 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5372999999999999e-05 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5096 3.b e34/23 3:d7ac1647:::rbd_object_map.112d64676e7f:head v 34'89, mlcod=34'85)", 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.779966+0000", 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.4846533600000003, 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005475972, 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.779966+0000", 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.779966+0000", 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.779967+0000", 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6199999999999997e-07 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.779975+0000", 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9719999999999992e-06 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.779992+0000", 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7825e-05 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.780001+0000", 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1920000000000005e-06 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.780084+0000", 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3251999999999995e-05 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.780121+0000", 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7666999999999997e-05 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785399+0000", 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052775519999999996 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785442+0000", 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2750000000000002e-05 2024-09-16T13:17:13.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5102 3.b e34/23 3:d7ac1647:::rbd_object_map.112d64676e7f:head v 34'95, mlcod=34'87)", 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.787413+0000", 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.4772063879999999, 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046315050000000002, 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787413+0000", 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787413+0000", 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787414+0000", 2024-09-16T13:17:13.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.122e-06 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787424+0000", 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6199999999999994e-06 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787442+0000", 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8522e-05 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787451+0000", 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4620000000000002e-06 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787504+0000", 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2348000000000001e-05 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787523+0000", 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8906000000000002e-05 2024-09-16T13:17:13.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792000+0000", 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044775120000000003 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792044+0000", 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4013e-05 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5103 3.b e34/23 3:d7ac1647:::rbd_object_map.112d64676e7f:head v 34'96, mlcod=34'88)", 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.788560+0000", 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.4760595849999998, 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005971021, 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788560+0000", 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788560+0000", 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788560+0000", 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0699999999999996e-07 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788566+0000", 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1909999999999997e-06 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788578+0000", 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1090999999999999e-05 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788583+0000", 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7039999999999999e-06 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788615+0000", 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2087e-05 2024-09-16T13:17:13.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788634+0000", 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8749999999999998e-05 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794491+0000", 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058572620000000002 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794531+0000", 2024-09-16T13:17:13.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9229e-05 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5104 3.b e34/23 3:d7ac1647:::rbd_object_map.112d64676e7f:head v 34'97, mlcod=34'88)", 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.789267+0000", 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.4753516710000003, 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052896729999999999, 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789267+0000", 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789267+0000", 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789268+0000", 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789276+0000", 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3229999999999998e-06 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789290+0000", 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3730999999999999e-05 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789296+0000", 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.48e-06 2024-09-16T13:17:13.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789342+0000", 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6662999999999997e-05 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789369+0000", 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7155999999999999e-05 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794533+0000", 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051634990000000002 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794557+0000", 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4213e-05 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6767 3.1 3:8190419c:::rbd_data.112d6fb8d387.00000000000002e3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.067087+0000", 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.1975324359999999, 2024-09-16T13:17:13.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054159819999999997, 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6767 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.067087+0000", 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.067087+0000", 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.067088+0000", 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7719999999999999e-06 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.067098+0000", 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9620000000000005e-06 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.067123+0000", 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4218000000000001e-05 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.067134+0000", 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1800000000000001e-05 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.067208+0000", 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3149e-05 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.067309+0000", 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010185500000000001 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.067477+0000", 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00016755599999999999 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.069332+0000", 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018547839999999999 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.072366+0000", 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030345699999999999 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.072452+0000", 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5903999999999996e-05 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.072503+0000", 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0411999999999998e-05 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6846 3.6 3:6577d1c9:::rbd_data.112d6fb8d387.0000000000000342:head [set-alloc-hint object_size 262144 write_size 262144,write 183922~38816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.243676+0000", 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.020943028, 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0076044620000000002, 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6846 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.243676+0000", 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.243676+0000", 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.243678+0000", 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1100000000000001e-06 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.243704+0000", 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5925999999999999e-05 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.243746+0000", 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2148999999999999e-05 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.243756+0000", 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3780000000000007e-06 2024-09-16T13:17:13.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.243860+0000", 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104337 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.243871+0000", 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0536e-05 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.248157+0000", 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042861280000000002 2024-09-16T13:17:13.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.248337+0000", 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0001806 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.248562+0000", 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000224818 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.250449+0000", 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001887131 2024-09-16T13:17:13.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.251142+0000", 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069268200000000002 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.251225+0000", 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3337999999999994e-05 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.251281+0000", 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5328999999999998e-05 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.369 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7036 3.6 3:638814d6:::rbd_data.112d6fb8d387.00000000000002f2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.692123+0000", 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.57249626600000003, 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046754140000000001, 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7036 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692123+0000", 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692123+0000", 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692127+0000", 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9600000000000002e-06 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692143+0000", 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6500000000000001e-05 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692195+0000", 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1533000000000003e-05 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692215+0000", 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0389000000000001e-05 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692301+0000", 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5947999999999998e-05 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692427+0000", 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000125967 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692558+0000", 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013097400000000001 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.696112+0000", 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035539959999999998 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.696542+0000", 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042950200000000001 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.696699+0000", 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000157417 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.696798+0000", 2024-09-16T13:17:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9228000000000006e-05 2024-09-16T13:17:13.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.376 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.376 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.376 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:13.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:13.378 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #4, image image_client.0-clone4 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent5 2024-09-16T13:17:13.379 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:13.370+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:13.478 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:13.470+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:13.487 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:13.487 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:13.487 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:13.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:13.579 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:13.574+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:13.602 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:13.602 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:13.603 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:13.603 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:13.603 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:13.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:13.680 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:13.672+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:13.720 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2439 3.0 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007802+0000", 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.704710875, 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065199539999999997, 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.721 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2439 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007802+0000", 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007802+0000", 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007803+0000", 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007810+0000", 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6359999999999997e-06 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007830+0000", 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9922999999999999e-05 2024-09-16T13:17:13.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007838+0000", 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6839999999999996e-06 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007901+0000", 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2963e-05 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007980+0000", 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9871999999999997e-05 2024-09-16T13:17:13.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008084+0000", 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103194 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011021+0000", 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029373279999999999 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014209+0000", 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003187984 2024-09-16T13:17:13.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014263+0000", 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3801999999999998e-05 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014322+0000", 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9151000000000003e-05 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.726 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2442 3.0 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009058+0000", 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.7034550680000002, 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054328409999999999, 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2442 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009058+0000", 2024-09-16T13:17:13.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009058+0000", 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009060+0000", 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.812e-06 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009068+0000", 2024-09-16T13:17:13.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6049999999999998e-06 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009098+0000", 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0137e-05 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009109+0000", 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0326000000000001e-05 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009157+0000", 2024-09-16T13:17:13.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8609999999999997e-05 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009209+0000", 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1974000000000001e-05 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009275+0000", 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5918000000000001e-05 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014018+0000", 2024-09-16T13:17:13.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047432380000000003 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014392+0000", 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037396599999999999 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014450+0000", 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7856000000000003e-05 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014491+0000", 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0398999999999997e-05 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2443 3.19 3:98eb4bee:::rbd_data.112dbfd5d823.00000000000000be:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009147+0000", 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.7033661700000007, 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050841319999999999, 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.732 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2443 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009147+0000", 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009147+0000", 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009147+0000", 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9100000000000002e-07 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009151+0000", 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5219999999999999e-06 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009158+0000", 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3019999999999997e-06 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009162+0000", 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0049999999999998e-06 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009221+0000", 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8718999999999998e-05 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009288+0000", 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6509999999999998e-05 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009398+0000", 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011022399999999999 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013996+0000", 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004597996 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014139+0000", 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014283699999999999 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014189+0000", 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0053999999999999e-05 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014231+0000", 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2072e-05 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2551 3.4 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.039408+0000", 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.6731044340000008, 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044004910000000003, 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2551 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039408+0000", 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039408+0000", 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039410+0000", 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.807e-06 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039511+0000", 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000100733 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039537+0000", 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6471000000000001e-05 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039545+0000", 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4089999999999997e-06 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039663+0000", 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118709 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039924+0000", 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026065000000000002 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040275+0000", 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035070000000000001 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043137+0000", 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00286207 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043662+0000", 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052466800000000005 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043763+0000", 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101223 2024-09-16T13:17:13.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043809+0000", 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6050999999999997e-05 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2552 3.4 3:25c42251:::rbd_data.112dbfd5d823.0000000000000019:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.039561+0000", 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.6729515500000005, 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047187289999999996, 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2552 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039561+0000", 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039561+0000", 2024-09-16T13:17:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039562+0000", 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0300000000000001e-06 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039567+0000", 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0170000000000004e-06 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039575+0000", 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9349999999999994e-06 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2490000000000001e-06 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.041753+0000", 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021746449999999998 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.042109+0000", 2024-09-16T13:17:13.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000355478 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.042593+0000", 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048401100000000001 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043852+0000", 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012594940000000001 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.044168+0000", 2024-09-16T13:17:13.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031550000000000003 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.044225+0000", 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7469999999999997e-05 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.044280+0000", 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.49e-05 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5115 3.4 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 163840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.790943+0000", 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.9215702270000001, 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038759340000000001, 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5115 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790943+0000", 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790943+0000", 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790944+0000", 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1069999999999999e-06 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790962+0000", 2024-09-16T13:17:13.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8199e-05 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790978+0000", 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6354999999999998e-05 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790986+0000", 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791033+0000", 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7426000000000003e-05 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791152+0000", 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118742 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791342+0000", 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000190476 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794215+0000", 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00287273 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794727+0000", 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00051181499999999999 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794780+0000", 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2941e-05 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794818+0000", 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8719e-05 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6751 3.0 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.057842+0000", 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.654670912, 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0075991560000000001, 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.754 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6751 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057842+0000", 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057842+0000", 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057845+0000", 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8059999999999999e-06 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057857+0000", 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2262999999999999e-05 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057908+0000", 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0998999999999997e-05 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057927+0000", 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.942e-05 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058082+0000", 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015499900000000001 2024-09-16T13:17:13.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058212+0000", 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012970399999999999 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058471+0000", 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025881500000000002 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.060170+0000", 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016994449999999999 2024-09-16T13:17:13.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.065330+0000", 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051600409999999998 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.065391+0000", 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0940999999999998e-05 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.065441+0000", 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:17:13.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6804 3.0 3:0753d729:::rbd_data.112d6fb8d387.0000000000000350:head [set-alloc-hint object_size 262144 write_size 262144,write 32625~14153] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.126137+0000", 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.5863759640000001, 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057116850000000002, 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.760 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6804 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126137+0000", 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126137+0000", 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126140+0000", 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5200000000000002e-06 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126166+0000", 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5876000000000001e-05 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126203+0000", 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7194000000000003e-05 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126216+0000", 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2768e-05 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126302+0000", 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6267000000000006e-05 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126313+0000", 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0519e-05 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.129024+0000", 2024-09-16T13:17:13.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002711326 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.129227+0000", 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020229300000000001 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.129453+0000", 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000226651 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.130858+0000", 2024-09-16T13:17:13.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014047249999999999 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.131719+0000", 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00086058599999999999 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.131798+0000", 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9147000000000006e-05 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.131848+0000", 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0813000000000003e-05 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6815 3.0 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head [set-alloc-hint object_size 262144 write_size 262144,write 204244~50197] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.148616+0000", 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.56389646, 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003881757, 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6815 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148616+0000", 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148616+0000", 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148619+0000", 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8509999999999999e-06 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148657+0000", 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8127999999999998e-05 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148691+0000", 2024-09-16T13:17:13.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4116000000000003e-05 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148708+0000", 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6934000000000001e-05 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148850+0000", 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014122599999999999 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149116+0000", 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000266247 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149351+0000", 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023480800000000001 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.151081+0000", 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017300519999999999 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.152268+0000", 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011874660000000001 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.152394+0000", 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012548099999999999 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.152498+0000", 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104448 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6938 3.4 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head [set-alloc-hint object_size 262144 write_size 262144,write 88878~42194] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.463722+0000", 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.248790412, 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037018239999999998, 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6938 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463722+0000", 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463722+0000", 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463725+0000", 2024-09-16T13:17:13.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7369999999999998e-06 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463768+0000", 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3393000000000003e-05 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463808+0000", 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9573999999999998e-05 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463825+0000", 2024-09-16T13:17:13.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7237000000000001e-05 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463940+0000", 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114887 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.464277+0000", 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033724500000000001 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.464706+0000", 2024-09-16T13:17:13.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042812500000000001 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.466299+0000", 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015934269999999999 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.467274+0000", 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00097460199999999998 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.467361+0000", 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7571999999999994e-05 2024-09-16T13:17:13.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.467424+0000", 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6963 3.4 3:2713f365:::rbd_data.112d6fb8d387.0000000000000133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.554432+0000", 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.1580811959999999, 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043955720000000004, 2024-09-16T13:17:13.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6963 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554432+0000", 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554432+0000", 2024-09-16T13:17:13.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554434+0000", 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2639999999999998e-06 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554446+0000", 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2099000000000001e-05 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554470+0000", 2024-09-16T13:17:13.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.425e-05 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554479+0000", 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4230000000000005e-06 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554598+0000", 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011938399999999999 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554822+0000", 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000224455 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555113+0000", 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029075699999999999 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557037+0000", 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00192352 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558750+0000", 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017129420000000001 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558791+0000", 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1159999999999999e-05 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558827+0000", 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6318e-05 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.782 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6964 3.0 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.554508+0000", 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.158004383, 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067699020000000004, 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6964 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.783 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554508+0000", 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554508+0000", 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554509+0000", 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554512+0000", 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8569999999999999e-06 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554517+0000", 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3179999999999997e-06 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554520+0000", 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1350000000000001e-06 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554610+0000", 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9963000000000006e-05 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554822+0000", 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021162199999999999 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555113+0000", 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000291367 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558708+0000", 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035949939999999998 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.561145+0000", 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002436723 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.561220+0000", 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5523000000000002e-05 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.561278+0000", 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7840000000000002e-05 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6966 3.0 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:13.788 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.1579814639999999, 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095509229999999994, 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6966 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:13.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554532+0000", 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2000000000000001e-07 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554534+0000", 2024-09-16T13:17:13.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9850000000000001e-06 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554537+0000", 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.664e-06 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554541+0000", 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2909999999999999e-06 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557170+0000", 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026295310000000001 2024-09-16T13:17:13.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557345+0000", 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017473200000000001 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557618+0000", 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027282800000000002 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563504+0000", 2024-09-16T13:17:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058862649999999999 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563923+0000", 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000419004 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564003+0000", 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0072000000000002e-05 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564082+0000", 2024-09-16T13:17:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9231000000000001e-05 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6969 3.4 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557236+0000", 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.1552765469999999, 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0068390430000000004, 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6969 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557236+0000", 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557236+0000", 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557237+0000", 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.094e-06 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557241+0000", 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.968e-06 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557255+0000", 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3511e-05 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557261+0000", 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557311+0000", 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9913999999999997e-05 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557375+0000", 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4010999999999994e-05 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557479+0000", 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104208 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560902+0000", 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034233449999999999 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564019+0000", 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031162260000000002 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564048+0000", 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9711000000000001e-05 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564075+0000", 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6948999999999999e-05 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7034 3.19 3:9ef0f110:::rbd_data.112d6fb8d387.0000000000000211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:13.799 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.691985+0000", 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.0205281980000001, 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049065949999999997, 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7034 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.691985+0000", 2024-09-16T13:17:13.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.691985+0000", 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.691988+0000", 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7359999999999999e-06 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692005+0000", 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7042999999999999e-05 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692050+0000", 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4425999999999998e-05 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692067+0000", 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-05 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692208+0000", 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014077299999999999 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692376+0000", 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0001682 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692645+0000", 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000269597 2024-09-16T13:17:13.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.693675+0000", 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010301170000000001 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.696758+0000", 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030828700000000001 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:13.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.696828+0000", 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9764000000000005e-05 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.696891+0000", 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3069000000000002e-05 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7166 3.4 3:249f6365:::rbd_data.112d64676e7f.0000000000000008:8 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:13.805 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:13.206764+0000", 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.50574863599999997, 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003713643, 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7166 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206764+0000", 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206764+0000", 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206765+0000", 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.78e-07 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206770+0000", 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9520000000000002e-06 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206780+0000", 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0010000000000001e-05 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206785+0000", 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9150000000000004e-06 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210127+0000", 2024-09-16T13:17:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003341789 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210254+0000", 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012760899999999999 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210478+0000", 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022338999999999999 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7169 3.4 3:249f6365:::rbd_data.112d64676e7f.0000000000000008:8 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:13.206790+0000", 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.50572288099999996, 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040801869999999999, 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7169 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206790+0000", 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206790+0000", 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206791+0000", 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.09e-07 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206793+0000", 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5780000000000001e-06 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206801+0000", 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0620000000000001e-06 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206806+0000", 2024-09-16T13:17:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1999999999999996e-06 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210529+0000", 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037235160000000001 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210657+0000", 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000128131 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210870+0000", 2024-09-16T13:17:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021289099999999999 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7175 3.4 3:2361a783:::rbd_data.112d64676e7f.000000000000000e:8 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:13.206810+0000", 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.505702859, 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004513563, 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7175 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206810+0000", 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206810+0000", 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206810+0000", 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8699999999999995e-07 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206813+0000", 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4279999999999999e-06 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206820+0000", 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6800000000000004e-06 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206826+0000", 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.686e-06 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210941+0000", 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041142729999999999 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.211095+0000", 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000154752 2024-09-16T13:17:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.211323+0000", 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022815699999999999 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7181 3.4 3:2361a783:::rbd_data.112d64676e7f.000000000000000e:8 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:13.206831+0000", 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.505682042, 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048806450000000003, 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7181 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206831+0000", 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206831+0000", 2024-09-16T13:17:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206831+0000", 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1999999999999999e-07 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206837+0000", 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3970000000000002e-06 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206843+0000", 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7379999999999997e-06 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206847+0000", 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.771e-06 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.211381+0000", 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045338740000000002 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.211510+0000", 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000128398 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.211711+0000", 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020184700000000001 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7191 3.4 3:249f6365:::rbd_data.112d64676e7f.0000000000000008:8 [read 163840~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:13.208632+0000", 2024-09-16T13:17:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.50388049300000004, 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037631119999999999, 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7191 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208632+0000", 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208632+0000", 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208633+0000", 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5099999999999998e-07 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208638+0000", 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0599999999999998e-06 2024-09-16T13:17:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208651+0000", 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2480000000000001e-05 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208670+0000", 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9021e-05 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.212214+0000", 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035438969999999998 2024-09-16T13:17:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.212288+0000", 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4683000000000003e-05 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.212395+0000", 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010702 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:13.827 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:13.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:13.828 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:13.820+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:13.916 INFO:teuthology.orchestra.run.smithi098.stdout:1267 trunc from 0x7ce9c20 to 0x4af04ca 2024-09-16T13:17:13.928 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:13.922+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:13.963 INFO:teuthology.orchestra.run.smithi098.stdout:1269 read 0x88ab00 thru 0x8994eb (0xe9ec bytes) 2024-09-16T13:17:13.966 INFO:teuthology.orchestra.run.smithi098.stdout:1270 trunc from 0x4af04ca to 0x3b3cd41 2024-09-16T13:17:13.975 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:13.975 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:13.975 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:13.975 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:13.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:14.010 INFO:teuthology.orchestra.run.smithi098.stdout:1272 read 0x39a7ad thru 0x39f0b6 (0x490a bytes) 2024-09-16T13:17:14.011 INFO:teuthology.orchestra.run.smithi098.stdout:1273 writesame 0x25b0a1e thru 0x25b587b data_size 0x7(0x4e5e bytes) 2024-09-16T13:17:14.029 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:14.024+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:14.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:14.059 INFO:teuthology.orchestra.run.smithi098.stdout:1274 write 0xd65f95f thru 0xd66b688 (0xbd2a bytes) 2024-09-16T13:17:14.064 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:14.064 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:14.064 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-16T13:17:14.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-16T13:17:14.084 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:14.084 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:14.084 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:14.084 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:14.085 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:14.088 INFO:teuthology.orchestra.run.smithi098.stdout:1275 write 0x961e55d thru 0x962d56e (0xf012 bytes) 2024-09-16T13:17:14.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:14.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:17:14.110 INFO:teuthology.orchestra.run.smithi098.stdout:1276 punch from 0xbaa54c3 to 0xbaae6fd, (0x923a bytes) 2024-09-16T13:17:14.110 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:14.129 INFO:teuthology.orchestra.run.smithi098.stdout:1280 trunc from 0xd66b689 to 0x23b3647 2024-09-16T13:17:14.130 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:14.126+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:14.173 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:14.173 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:14.199 INFO:teuthology.orchestra.run.smithi098.stdout:1283 writesame 0xb181841 thru 0xb18ed68 data_size 0x4c(0xd528 bytes) 2024-09-16T13:17:14.199 INFO:teuthology.orchestra.run.smithi098.stdout:1284 write 0x5a45862 thru 0x5a4b664 (0x5e03 bytes) 2024-09-16T13:17:14.208 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:14.222 INFO:teuthology.orchestra.run.smithi098.stdout:1285 trunc from 0xb18ed69 to 0x75cee5a 2024-09-16T13:17:14.230 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:14.224+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:14.235 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:14.235 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:14.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:14.235 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:14.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.235 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1264 3.4 e27/23 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head v 27'4, mlcod=26'2)", 2024-09-16T13:17:14.235 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.727400398, 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026410209999999999, 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.236 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501250+0000", 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2410000000000001e-06 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501259+0000", 2024-09-16T13:17:14.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501277+0000", 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8683999999999999e-05 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501288+0000", 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.065e-05 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501562+0000", 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027421200000000002 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501618+0000", 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6045000000000003e-05 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503841+0000", 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022230829999999998 2024-09-16T13:17:14.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503890+0000", 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8526000000000003e-05 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.22125671, 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:17:14.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.220471211, 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:14.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:17:14.249 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.219304289, 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.216429657999999, 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:14.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:17:14.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:17:14.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.213415918000001, 2024-09-16T13:17:14.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:17:14.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.188211715, 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.263 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:17:14.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.4220628259999994, 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:14.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:17:14.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:17:14.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5115 3.4 e34/23 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head v 34'14, mlcod=31'12)", 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.4370147199999996, 2024-09-16T13:17:14.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029400680000000001, 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791635+0000", 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4200000000000005e-07 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791643+0000", 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.157e-06 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791660+0000", 2024-09-16T13:17:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6246e-05 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791666+0000", 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791743+0000", 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7000000000000001e-05 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791756+0000", 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2729e-05 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794523+0000", 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027675949999999999 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794575+0000", 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1082000000000003e-05 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.276 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5290 3.9 e34/23 3:975e9c33:::rbd_data.112d64676e7f.00000000000000b8:head v 34'26, mlcod=32'24)", 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.958105+0000", 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.2705443020000002, 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002485054, 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958105+0000", 2024-09-16T13:17:14.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958105+0000", 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958106+0000", 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2139999999999999e-06 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958117+0000", 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1188e-05 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958143+0000", 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5670999999999999e-05 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958152+0000", 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8389999999999996e-06 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958204+0000", 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2219999999999998e-05 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958226+0000", 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2334e-05 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960538+0000", 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023112779999999999 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960590+0000", 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2309999999999999e-05 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5435 3.4 e34/23 3:248e1f52:::rbd_data.112d64676e7f.0000000000000036:head v 34'21, mlcod=34'19)", 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:08.157555+0000", 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.071094156, 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024120550000000002, 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.281 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157555+0000", 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157555+0000", 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157556+0000", 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209999999999999e-06 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157590+0000", 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3918000000000003e-05 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157608+0000", 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8059000000000001e-05 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157615+0000", 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7129999999999999e-06 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157662+0000", 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7250000000000003e-05 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157678+0000", 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5577999999999999e-05 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.159928+0000", 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002250087 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.159967+0000", 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9329000000000002e-05 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.1699605229999999, 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:17:14.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.169351083, 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:14.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:17:14.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:17:14.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:17:14.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6815 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'44, mlcod=36'42)", 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.149529+0000", 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.079119945, 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024510030000000002, 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149529+0000", 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149529+0000", 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149531+0000", 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0530000000000001e-06 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149642+0000", 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110951 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149685+0000", 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3075999999999999e-05 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149704+0000", 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8592000000000001e-05 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149818+0000", 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113842 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149876+0000", 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8522e-05 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.151895+0000", 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020183179999999998 2024-09-16T13:17:14.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.151980+0000", 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5649000000000006e-05 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6864 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'46, mlcod=37'44)", 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.300990+0000", 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.9276590279999999, 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022853959999999999, 2024-09-16T13:17:14.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.300990+0000", 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.300990+0000", 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.300992+0000", 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0439999999999998e-06 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.301014+0000", 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2166000000000001e-05 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.301051+0000", 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6891999999999998e-05 2024-09-16T13:17:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.301064+0000", 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.291e-05 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.301188+0000", 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012352000000000001 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.301273+0000", 2024-09-16T13:17:14.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5600999999999996e-05 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.303145+0000", 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018716760000000001 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.303276+0000", 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013058699999999999 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6963 3.4 e37/23 3:2713f365:::rbd_data.112d6fb8d387.0000000000000133:head v 37'31, mlcod=37'29)", 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.672786933, 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027598190000000002, 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555864+0000", 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555882+0000", 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7816e-05 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555909+0000", 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7838999999999999e-05 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555923+0000", 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3577000000000001e-05 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556083+0000", 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015987299999999999 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556132+0000", 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8655000000000001e-05 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558539+0000", 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024076969999999999 2024-09-16T13:17:14.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558622+0000", 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2777999999999999e-05 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.672067269, 2024-09-16T13:17:14.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.6709664530000001, 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:14.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:17:14.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.6708268130000001, 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.317 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:17:14.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:17:14.321 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.6703262830000001, 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:14.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:17:14.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:14.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:14.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:17:14.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:17:14.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:14.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:14.326 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:14.327 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:14.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:14.328 INFO:teuthology.orchestra.run.smithi098.stdout:1286 write 0x9b77748 thru 0x9b81c91 (0xa54a bytes) 2024-09-16T13:17:14.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-16T13:17:14.329 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:14.330 INFO:teuthology.orchestra.run.smithi098.stdout:1287 write 0x8571f75 thru 0x857c798 (0xa824 bytes) 2024-09-16T13:17:14.330 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:14.331 INFO:teuthology.orchestra.run.smithi098.stdout:1288 read 0x20c4c79 thru 0x20cbe05 (0x718d bytes) 2024-09-16T13:17:14.331 INFO:teuthology.orchestra.run.smithi098.stdout:1291 read 0x64755fd thru 0x647675d (0x1161 bytes) 2024-09-16T13:17:14.331 INFO:teuthology.orchestra.run.smithi098.stdout:1292 read 0x538fd07 thru 0x539f6a0 (0xf99a bytes) 2024-09-16T13:17:14.331 INFO:teuthology.orchestra.run.smithi098.stdout:1293 writesame 0xca575f5 thru 0xca5f2a1 data_size 0x1(0x7cad bytes) 2024-09-16T13:17:14.331 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:14.331 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:14.331 INFO:teuthology.orchestra.run.smithi098.stdout:1297 punch from 0x84670e9 to 0x846b107, (0x401e bytes) 2024-09-16T13:17:14.331 INFO:teuthology.orchestra.run.smithi098.stdout:1298 writesame 0x4485248 thru 0x449221b data_size 0x4(0xcfd4 bytes) 2024-09-16T13:17:14.332 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:14.332 INFO:teuthology.orchestra.run.smithi098.stdout:1300 read 0x318d06b thru 0x3197b0f (0xaaa5 bytes) 2024-09-16T13:17:14.332 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:14.333 INFO:teuthology.orchestra.run.smithi098.stdout:1301 write 0x5059215 thru 0x505d68a (0x4476 bytes) 2024-09-16T13:17:14.335 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:14.330+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:14.341 INFO:teuthology.orchestra.run.smithi098.stdout:1302 writesame 0x8e3402f thru 0x8e39046 data_size 0x16(0x5018 bytes) 2024-09-16T13:17:14.351 INFO:teuthology.orchestra.run.smithi098.stdout:1303 writesame 0x51cc0e6 thru 0x51d858a data_size 0x1(0xc4a5 bytes) 2024-09-16T13:17:14.359 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:14.386 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:14.402 INFO:teuthology.orchestra.run.smithi098.stdout:1305 punch from 0x8f2b99d to 0x8f32625, (0x6c88 bytes) 2024-09-16T13:17:14.410 INFO:teuthology.orchestra.run.smithi098.stdout:1306 trunc from 0xca5f2a2 to 0x7b84b2a 2024-09-16T13:17:14.435 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:14.430+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:14.438 INFO:teuthology.orchestra.run.smithi098.stdout:1307 write 0x12d0c7f thru 0x12d1dba (0x113c bytes) 2024-09-16T13:17:14.444 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:14.445 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:14.448 INFO:teuthology.orchestra.run.smithi098.stdout:1308 clone 7 order 20 su 262144 sc 4 2024-09-16T13:17:14.455 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:14.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-16T13:17:14.536 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:14.530+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:14.622 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:14.622 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:14.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:14.637 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:14.632+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:14.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-16T13:17:14.726 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:14.726 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:14.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:14.738 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:14.732+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:14.839 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:14.834+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:14.841 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:14.841 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:14.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:14.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-16T13:17:14.941 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:14.936+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:14.955 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:14.955 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:14.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:15.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-16T13:17:15.041 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:15.034+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:15.061 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:15.061 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:15.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:15.103 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:17:15.142 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:15.138+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:15.182 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:15.182 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:15.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-09-16T13:17:15.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:15.206 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone6 from 0x7b84b2a (overlap 0x23b3647) to 0x0 2024-09-16T13:17:15.243 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:15.236+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:15.329 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:15.329 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:15.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:15.344 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:15.338+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:15.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-09-16T13:17:15.424 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #5, image image_client.0-clone5 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent6 2024-09-16T13:17:15.432 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:15.432 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:15.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:15.444 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:15.440+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:15.546 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:15.540+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:15.547 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:15.547 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:15.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:15.646 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:15.642+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:15.650 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:15.651 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:15.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:15.695 INFO:teuthology.orchestra.run.smithi098.stdout:1309 writesame 0x311578c thru 0x312468f data_size 0x6(0xef04 bytes) 2024-09-16T13:17:15.721 INFO:teuthology.orchestra.run.smithi098.stdout:1310 writesame 0x7a99fcc thru 0x7aa3e45 data_size 0x5(0x9e7a bytes) 2024-09-16T13:17:15.747 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:15.742+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:15.760 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:15.760 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:15.762 INFO:teuthology.orchestra.run.smithi098.stdout:1311 write 0xe042c8e thru 0xe04ce8f (0xa202 bytes) 2024-09-16T13:17:15.769 INFO:teuthology.orchestra.run.smithi098.stdout:1314 write 0x1a48066 thru 0x1a52983 (0xa91e bytes) 2024-09-16T13:17:15.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:15.779 INFO:teuthology.orchestra.run.smithi098.stdout:1315 read 0x42625f3 thru 0x4270b40 (0xe54e bytes) 2024-09-16T13:17:15.781 INFO:teuthology.orchestra.run.smithi098.stdout:1316 read 0x3abb29d thru 0x3abc9e7 (0x174b bytes) 2024-09-16T13:17:15.782 INFO:teuthology.orchestra.run.smithi098.stdout:1317 read 0x5e3411e thru 0x5e3ec32 (0xab15 bytes) 2024-09-16T13:17:15.782 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:15.782 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:15.782 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:15.782 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:15.782 INFO:teuthology.orchestra.run.smithi098.stdout:1324 write 0x8aec376 thru 0x8af62d3 (0x9f5e bytes) 2024-09-16T13:17:15.787 INFO:teuthology.orchestra.run.smithi098.stdout:1325 write 0xca76ea9 thru 0xca7ef41 (0x8099 bytes) 2024-09-16T13:17:15.792 INFO:teuthology.orchestra.run.smithi098.stdout:1327 write 0x6bcf1cb thru 0x6bd4f2f (0x5d65 bytes) 2024-09-16T13:17:15.798 INFO:teuthology.orchestra.run.smithi098.stdout:1328 read 0x2bbc60c thru 0x2bc9e0b (0xd800 bytes) 2024-09-16T13:17:15.799 INFO:teuthology.orchestra.run.smithi098.stdout:1329 read 0x668bdcc thru 0x6693484 (0x76b9 bytes) 2024-09-16T13:17:15.822 INFO:teuthology.orchestra.run.smithi098.stdout:1330 trunc from 0xe04ce90 to 0x1bfed4c 2024-09-16T13:17:15.848 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:15.844+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:15.851 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:15.868 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:15.868 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:15.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:15.884 INFO:teuthology.orchestra.run.smithi098.stdout:1332 writesame 0xe1e0e05 thru 0xe1e3e88 data_size 0x12(0x3084 bytes) 2024-09-16T13:17:15.889 INFO:teuthology.orchestra.run.smithi098.stdout:1335 trunc from 0xe1e3e89 to 0xcbe3163 2024-09-16T13:17:15.899 INFO:teuthology.orchestra.run.smithi098.stdout:1336 punch from 0x1d11a70 to 0x1d1d120, (0xb6b0 bytes) 2024-09-16T13:17:15.899 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:15.909 INFO:teuthology.orchestra.run.smithi098.stdout:1339 trunc from 0xcbe3163 to 0x6da9a78 2024-09-16T13:17:15.948 INFO:teuthology.orchestra.run.smithi098.stdout:1340 writesame 0xde0ff34 thru 0xde10893 data_size 0xf(0x960 bytes) 2024-09-16T13:17:15.948 INFO:teuthology.orchestra.run.smithi098.stdout:1341 read 0x22fe949 thru 0x23017da (0x2e92 bytes) 2024-09-16T13:17:15.948 INFO:teuthology.orchestra.run.smithi098.stdout:1342 read 0x76916ee thru 0x769da46 (0xc359 bytes) 2024-09-16T13:17:15.949 INFO:teuthology.orchestra.run.smithi098.stdout:1344 writesame 0x3b3b459 thru 0x3b404f8 data_size 0xa(0x50a0 bytes) 2024-09-16T13:17:15.950 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:15.946+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:15.958 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:15.959 INFO:teuthology.orchestra.run.smithi098.stdout:1346 read 0xe70eee thru 0xe74b57 (0x3c6a bytes) 2024-09-16T13:17:15.960 INFO:teuthology.orchestra.run.smithi098.stdout:1347 punch from 0x65caf99 to 0x65d5562, (0xa5c9 bytes) 2024-09-16T13:17:15.960 INFO:teuthology.orchestra.run.smithi098.stdout:1349 write 0x12e953d thru 0x12f65eb (0xd0af bytes) 2024-09-16T13:17:15.973 INFO:teuthology.orchestra.run.smithi098.stdout:1353 writesame 0x40c4b3a thru 0x40cd980 data_size 0x47(0x8e47 bytes) 2024-09-16T13:17:15.977 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:15.977 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:15.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:15.990 INFO:teuthology.orchestra.run.smithi098.stdout:1354 trunc from 0xde10894 to 0xa0178f0 2024-09-16T13:17:16.002 INFO:teuthology.orchestra.run.smithi098.stdout:1355 write 0x3a9f607 thru 0x3aaa794 (0xb18e bytes) 2024-09-16T13:17:16.008 INFO:teuthology.orchestra.run.smithi098.stdout:1356 write 0x5239b99 thru 0x52414c0 (0x7928 bytes) 2024-09-16T13:17:16.017 INFO:teuthology.orchestra.run.smithi098.stdout:1357 read 0x70eb484 thru 0x70ec799 (0x1316 bytes) 2024-09-16T13:17:16.017 INFO:teuthology.orchestra.run.smithi098.stdout:1358 write 0x6a592de thru 0x6a5a915 (0x1638 bytes) 2024-09-16T13:17:16.023 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:16.023 INFO:teuthology.orchestra.run.smithi098.stdout:1361 punch from 0x3753173 to 0x376290f, (0xf79c bytes) 2024-09-16T13:17:16.023 INFO:teuthology.orchestra.run.smithi098.stdout:1362 write 0x70df18 thru 0x710b19 (0x2c02 bytes) 2024-09-16T13:17:16.051 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:16.046+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:16.059 INFO:teuthology.orchestra.run.smithi098.stdout:1363 trunc from 0xa0178f0 to 0x2f98b05 2024-09-16T13:17:16.080 INFO:teuthology.orchestra.run.smithi098.stdout:1365 punch from 0xd888d6 to 0xd8aea6, (0x25d0 bytes) 2024-09-16T13:17:16.086 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:16.086 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:16.093 INFO:teuthology.orchestra.run.smithi098.stdout:1367 trunc from 0x2f98b05 to 0xa05071e 2024-09-16T13:17:16.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:16.103 INFO:teuthology.orchestra.run.smithi098.stdout:1368 write 0x50f5670 thru 0x5101691 (0xc022 bytes) 2024-09-16T13:17:16.119 INFO:teuthology.orchestra.run.smithi098.stdout:1370 trunc from 0xa05071e to 0xdaa6bad 2024-09-16T13:17:16.130 INFO:teuthology.orchestra.run.smithi098.stdout:1371 write 0xa983c79 thru 0xa984a14 (0xd9c bytes) 2024-09-16T13:17:16.147 INFO:teuthology.orchestra.run.smithi098.stdout:1373 writesame 0xdace981 thru 0xdadcb5d data_size 0x1(0xe1dd bytes) 2024-09-16T13:17:16.147 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:16.147 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:16.147 INFO:teuthology.orchestra.run.smithi098.stdout:1378 punch from 0x61ce8e9 to 0x61d0df1, (0x2508 bytes) 2024-09-16T13:17:16.147 INFO:teuthology.orchestra.run.smithi098.stdout:1379 write 0x60bd0cf thru 0x60c3251 (0x6183 bytes) 2024-09-16T13:17:16.151 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:16.144+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:16.156 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:16.157 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:16.157 INFO:teuthology.orchestra.run.smithi098.stdout:1382 write 0x4456396 thru 0x445a3f7 (0x4062 bytes) 2024-09-16T13:17:16.163 INFO:teuthology.orchestra.run.smithi098.stdout:1384 punch from 0x84a93b to 0x855aea, (0xb1af bytes) 2024-09-16T13:17:16.163 INFO:teuthology.orchestra.run.smithi098.stdout:1385 write 0x5319ac0 thru 0x5326f94 (0xd4d5 bytes) 2024-09-16T13:17:16.166 INFO:teuthology.orchestra.run.smithi098.stdout:1386 writesame 0xcf6ec66 thru 0xcf788d7 data_size 0x96(0x9c72 bytes) 2024-09-16T13:17:16.186 INFO:teuthology.orchestra.run.smithi098.stdout:1387 trunc from 0xdadcb5e to 0x7206cc3 2024-09-16T13:17:16.198 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:16.198 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:16.206 INFO:teuthology.orchestra.run.smithi098.stdout:1389 punch from 0x2f0bee0 to 0x2f128ea, (0x6a0a bytes) 2024-09-16T13:17:16.206 INFO:teuthology.orchestra.run.smithi098.stdout:1390 read 0x3edc3ff thru 0x3edd2f5 (0xef7 bytes) 2024-09-16T13:17:16.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:1391 trunc from 0x7206cc3 to 0xca82a64 2024-09-16T13:17:16.224 INFO:teuthology.orchestra.run.smithi098.stdout:1392 write 0x1583f54 thru 0x1588b3b (0x4be8 bytes) 2024-09-16T13:17:16.234 INFO:teuthology.orchestra.run.smithi098.stdout:1394 write 0x2689eec thru 0x268d994 (0x3aa9 bytes) 2024-09-16T13:17:16.240 INFO:teuthology.orchestra.run.smithi098.stdout:1395 read 0x8d4dcc5 thru 0x8d56085 (0x83c1 bytes) 2024-09-16T13:17:16.241 INFO:teuthology.orchestra.run.smithi098.stdout:1397 writesame 0x595e09b thru 0x59631fb data_size 0x1(0x5161 bytes) 2024-09-16T13:17:16.252 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:16.244+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:16.264 INFO:teuthology.orchestra.run.smithi098.stdout:1398 write 0x6bd0339 thru 0x6bdde01 (0xdac9 bytes) 2024-09-16T13:17:16.273 INFO:teuthology.orchestra.run.smithi098.stdout:1399 writesame 0x9281c8 thru 0x92ed49 data_size 0x42(0x6b82 bytes) 2024-09-16T13:17:16.293 INFO:teuthology.orchestra.run.smithi098.stdout:1400 read 0x7645e0a thru 0x7647806 (0x19fd bytes) 2024-09-16T13:17:16.294 INFO:teuthology.orchestra.run.smithi098.stdout:1402 trunc from 0xca82a64 to 0xbfa228d 2024-09-16T13:17:16.306 INFO:teuthology.orchestra.run.smithi098.stdout:1403 writesame 0x2bfa010 thru 0x2c05250 data_size 0x7b(0xb241 bytes) 2024-09-16T13:17:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:1404 clone 8 order 23 su 2097152 sc 4 2024-09-16T13:17:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:16.316 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:16.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:16.369 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:16.364+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:16.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:16.454 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:16.450+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:16.473 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:16.473 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:16.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:16.535 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:16.555 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:16.550+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:16.562 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:16.576 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:16.576 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:16.576 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:16.577 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:16.580 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:16.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:16.601 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:16.625 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:16.648 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:16.655 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:16.648+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:16.676 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:16.678 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:16.679 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:16.679 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:16.679 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:16.679 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:16.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:16.704 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:16.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:16.707 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:16.757 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:16.750+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:16.784 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:16.784 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:16.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:16.784 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:16.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2439 3.0 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007802+0000", 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.770621032999999, 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065199539999999997, 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2439 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.785 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007802+0000", 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007802+0000", 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007803+0000", 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007810+0000", 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6359999999999997e-06 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007830+0000", 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9922999999999999e-05 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007838+0000", 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6839999999999996e-06 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007901+0000", 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2963e-05 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007980+0000", 2024-09-16T13:17:16.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9871999999999997e-05 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008084+0000", 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000103194 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011021+0000", 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029373279999999999 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014209+0000", 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003187984 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014263+0000", 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3801999999999998e-05 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014322+0000", 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9151000000000003e-05 2024-09-16T13:17:16.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2442 3.0 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009058+0000", 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.769365226, 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054328409999999999, 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.791 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2442 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009058+0000", 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009058+0000", 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009060+0000", 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.812e-06 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009068+0000", 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6049999999999998e-06 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009098+0000", 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0137e-05 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009109+0000", 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0326000000000001e-05 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009157+0000", 2024-09-16T13:17:16.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8609999999999997e-05 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009209+0000", 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1974000000000001e-05 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009275+0000", 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5918000000000001e-05 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014018+0000", 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047432380000000003 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014392+0000", 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037396599999999999 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014450+0000", 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7856000000000003e-05 2024-09-16T13:17:16.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014491+0000", 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0398999999999997e-05 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2443 3.19 3:98eb4bee:::rbd_data.112dbfd5d823.00000000000000be:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009147+0000", 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.769276328, 2024-09-16T13:17:16.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050841319999999999, 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2443 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009147+0000", 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009147+0000", 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009147+0000", 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9100000000000002e-07 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009151+0000", 2024-09-16T13:17:16.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5219999999999999e-06 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009158+0000", 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3019999999999997e-06 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009162+0000", 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0049999999999998e-06 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009221+0000", 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8718999999999998e-05 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009288+0000", 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6509999999999998e-05 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009398+0000", 2024-09-16T13:17:16.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011022399999999999 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013996+0000", 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004597996 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014139+0000", 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014283699999999999 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014189+0000", 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0053999999999999e-05 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.014231+0000", 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2072e-05 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.803 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2551 3.4 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.039408+0000", 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.739014592, 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044004910000000003, 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2551 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039408+0000", 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039408+0000", 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039410+0000", 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.807e-06 2024-09-16T13:17:16.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039511+0000", 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000100733 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039537+0000", 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6471000000000001e-05 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039545+0000", 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4089999999999997e-06 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039663+0000", 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118709 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039924+0000", 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00026065000000000002 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040275+0000", 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035070000000000001 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043137+0000", 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00286207 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043662+0000", 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00052466800000000005 2024-09-16T13:17:16.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043763+0000", 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101223 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043809+0000", 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6050999999999997e-05 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:2552 3.4 3:25c42251:::rbd_data.112dbfd5d823.0000000000000019:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.039561+0000", 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.738861708, 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047187289999999996, 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.810 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 2552 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039561+0000", 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039561+0000", 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039562+0000", 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0300000000000001e-06 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039567+0000", 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0170000000000004e-06 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039575+0000", 2024-09-16T13:17:16.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9349999999999994e-06 2024-09-16T13:17:16.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:17:16.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2490000000000001e-06 2024-09-16T13:17:16.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.041753+0000", 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021746449999999998 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.042109+0000", 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000355478 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.042593+0000", 2024-09-16T13:17:16.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048401100000000001 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043852+0000", 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012594940000000001 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.044168+0000", 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031550000000000003 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.044225+0000", 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7469999999999997e-05 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.044280+0000", 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.49e-05 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5115 3.4 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 163840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.790943+0000", 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 8.9874803849999996, 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038759340000000001, 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5115 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.817 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790943+0000", 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790943+0000", 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790944+0000", 2024-09-16T13:17:16.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1069999999999999e-06 2024-09-16T13:17:16.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790962+0000", 2024-09-16T13:17:16.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8199e-05 2024-09-16T13:17:16.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790978+0000", 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6354999999999998e-05 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790986+0000", 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4240000000000004e-06 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791033+0000", 2024-09-16T13:17:16.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7426000000000003e-05 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791152+0000", 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000118742 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791342+0000", 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000190476 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794215+0000", 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00287273 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794727+0000", 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00051181499999999999 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794780+0000", 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2941e-05 2024-09-16T13:17:16.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794818+0000", 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8719e-05 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6751 3.0 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.057842+0000", 2024-09-16T13:17:16.823 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.7205810699999997, 2024-09-16T13:17:16.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0075991560000000001, 2024-09-16T13:17:16.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.824 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.824 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.824 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.824 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6751 2024-09-16T13:17:16.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.824 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057842+0000", 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057842+0000", 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057845+0000", 2024-09-16T13:17:16.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8059999999999999e-06 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057857+0000", 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2262999999999999e-05 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057908+0000", 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0998999999999997e-05 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.057927+0000", 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.942e-05 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058082+0000", 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015499900000000001 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058212+0000", 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012970399999999999 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058471+0000", 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00025881500000000002 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.060170+0000", 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016994449999999999 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.065330+0000", 2024-09-16T13:17:16.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051600409999999998 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.065391+0000", 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0940999999999998e-05 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.065441+0000", 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9722999999999998e-05 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.829 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.830 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6804 3.0 3:0753d729:::rbd_data.112d6fb8d387.0000000000000350:head [set-alloc-hint object_size 262144 write_size 262144,write 32625~14153] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:16.830 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.126137+0000", 2024-09-16T13:17:16.830 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.6522861219999996, 2024-09-16T13:17:16.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057116850000000002, 2024-09-16T13:17:16.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.830 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6804 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126137+0000", 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126137+0000", 2024-09-16T13:17:16.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126140+0000", 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5200000000000002e-06 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126166+0000", 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5876000000000001e-05 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126203+0000", 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7194000000000003e-05 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126216+0000", 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2768e-05 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126302+0000", 2024-09-16T13:17:16.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6267000000000006e-05 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op must wait for map", 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.126313+0000", 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0519e-05 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.129024+0000", 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002711326 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.129227+0000", 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020229300000000001 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.129453+0000", 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000226651 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.130858+0000", 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014047249999999999 2024-09-16T13:17:16.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.131719+0000", 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00086058599999999999 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.131798+0000", 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9147000000000006e-05 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.131848+0000", 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0813000000000003e-05 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6815 3.0 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head [set-alloc-hint object_size 262144 write_size 262144,write 204244~50197] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.148616+0000", 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.6298066179999999, 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003881757, 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6815 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148616+0000", 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148616+0000", 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148619+0000", 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8509999999999999e-06 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148657+0000", 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8127999999999998e-05 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148691+0000", 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4116000000000003e-05 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148708+0000", 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6934000000000001e-05 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.148850+0000", 2024-09-16T13:17:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014122599999999999 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149116+0000", 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000266247 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149351+0000", 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023480800000000001 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.151081+0000", 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017300519999999999 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.152268+0000", 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011874660000000001 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.152394+0000", 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012548099999999999 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.152498+0000", 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104448 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6938 3.4 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head [set-alloc-hint object_size 262144 write_size 262144,write 88878~42194] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.463722+0000", 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3147005700000003, 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037018239999999998, 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.843 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6938 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463722+0000", 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463722+0000", 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463725+0000", 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7369999999999998e-06 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463768+0000", 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3393000000000003e-05 2024-09-16T13:17:16.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463808+0000", 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9573999999999998e-05 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463825+0000", 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7237000000000001e-05 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.463940+0000", 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114887 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.464277+0000", 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033724500000000001 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.464706+0000", 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042812500000000001 2024-09-16T13:17:16.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.466299+0000", 2024-09-16T13:17:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015934269999999999 2024-09-16T13:17:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.467274+0000", 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00097460199999999998 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.467361+0000", 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7571999999999994e-05 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.467424+0000", 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3025000000000001e-05 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.849 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6963 3.4 3:2713f365:::rbd_data.112d6fb8d387.0000000000000133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.554432+0000", 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.2239913539999998, 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043955720000000004, 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6963 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554432+0000", 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554432+0000", 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554434+0000", 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2639999999999998e-06 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554446+0000", 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2099000000000001e-05 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554470+0000", 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.425e-05 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554479+0000", 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4230000000000005e-06 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554598+0000", 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011938399999999999 2024-09-16T13:17:16.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554822+0000", 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000224455 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555113+0000", 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029075699999999999 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557037+0000", 2024-09-16T13:17:16.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00192352 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558750+0000", 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017129420000000001 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558791+0000", 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1159999999999999e-05 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558827+0000", 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6318e-05 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6964 3.0 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.554508+0000", 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.2239145410000001, 2024-09-16T13:17:16.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067699020000000004, 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6964 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554508+0000", 2024-09-16T13:17:16.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554508+0000", 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554509+0000", 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554512+0000", 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8569999999999999e-06 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554517+0000", 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3179999999999997e-06 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554520+0000", 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1350000000000001e-06 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554610+0000", 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9963000000000006e-05 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554822+0000", 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021162199999999999 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555113+0000", 2024-09-16T13:17:16.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000291367 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558708+0000", 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035949939999999998 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.561145+0000", 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002436723 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.561220+0000", 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5523000000000002e-05 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.561278+0000", 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7840000000000002e-05 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6966 3.0 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:16.862 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.223891622, 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095509229999999994, 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6966 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:16.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554532+0000", 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2000000000000001e-07 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554534+0000", 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9850000000000001e-06 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554537+0000", 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.664e-06 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554541+0000", 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2909999999999999e-06 2024-09-16T13:17:16.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557170+0000", 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026295310000000001 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557345+0000", 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017473200000000001 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557618+0000", 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027282800000000002 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563504+0000", 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058862649999999999 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563923+0000", 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000419004 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564003+0000", 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0072000000000002e-05 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564082+0000", 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9231000000000001e-05 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6969 3.4 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557236+0000", 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.221186705, 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0068390430000000004, 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6969 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557236+0000", 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557236+0000", 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557237+0000", 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.094e-06 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557241+0000", 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.968e-06 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557255+0000", 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3511e-05 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557261+0000", 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557311+0000", 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9913999999999997e-05 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557375+0000", 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4010999999999994e-05 2024-09-16T13:17:16.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557479+0000", 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104208 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560902+0000", 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034233449999999999 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564019+0000", 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031162260000000002 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564048+0000", 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9711000000000001e-05 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564075+0000", 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6948999999999999e-05 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.874 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.875 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.875 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7034 3.19 3:9ef0f110:::rbd_data.112d6fb8d387.0000000000000211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:16.875 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.691985+0000", 2024-09-16T13:17:16.875 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.0864383560000004, 2024-09-16T13:17:16.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0049065949999999997, 2024-09-16T13:17:16.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.875 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:16.875 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7034 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.691985+0000", 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.691985+0000", 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.691988+0000", 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7359999999999999e-06 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692005+0000", 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7042999999999999e-05 2024-09-16T13:17:16.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692050+0000", 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4425999999999998e-05 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692067+0000", 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-05 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692208+0000", 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014077299999999999 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692376+0000", 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0001682 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.692645+0000", 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000269597 2024-09-16T13:17:16.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.693675+0000", 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010301170000000001 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.696758+0000", 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030828700000000001 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.696828+0000", 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9764000000000005e-05 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.696891+0000", 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3069000000000002e-05 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7166 3.4 3:249f6365:::rbd_data.112d64676e7f.0000000000000008:8 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:13.206764+0000", 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.5716587940000002, 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003713643, 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7166 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206764+0000", 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206764+0000", 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206765+0000", 2024-09-16T13:17:16.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.78e-07 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206770+0000", 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9520000000000002e-06 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206780+0000", 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0010000000000001e-05 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206785+0000", 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9150000000000004e-06 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210127+0000", 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003341789 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210254+0000", 2024-09-16T13:17:16.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012760899999999999 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210478+0000", 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022338999999999999 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7169 3.4 3:249f6365:::rbd_data.112d64676e7f.0000000000000008:8 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:13.206790+0000", 2024-09-16T13:17:16.886 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.571633039, 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040801869999999999, 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7169 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206790+0000", 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206790+0000", 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206791+0000", 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.09e-07 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206793+0000", 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5780000000000001e-06 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206801+0000", 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0620000000000001e-06 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206806+0000", 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1999999999999996e-06 2024-09-16T13:17:16.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210529+0000", 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037235160000000001 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210657+0000", 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000128131 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210870+0000", 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00021289099999999999 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7175 3.4 3:2361a783:::rbd_data.112d64676e7f.000000000000000e:8 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:13.206810+0000", 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.5716130169999998, 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004513563, 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.891 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7175 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206810+0000", 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206810+0000", 2024-09-16T13:17:16.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206810+0000", 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8699999999999995e-07 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206813+0000", 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4279999999999999e-06 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206820+0000", 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6800000000000004e-06 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206826+0000", 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.686e-06 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.210941+0000", 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041142729999999999 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.211095+0000", 2024-09-16T13:17:16.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000154752 2024-09-16T13:17:16.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.211323+0000", 2024-09-16T13:17:16.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022815699999999999 2024-09-16T13:17:16.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.895 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7181 3.4 3:2361a783:::rbd_data.112d64676e7f.000000000000000e:8 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:13.206831+0000", 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.5715922, 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048806450000000003, 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.896 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7181 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206831+0000", 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206831+0000", 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206831+0000", 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1999999999999999e-07 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206837+0000", 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3970000000000002e-06 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206843+0000", 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7379999999999997e-06 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.206847+0000", 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.771e-06 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.211381+0000", 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045338740000000002 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.211510+0000", 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000128398 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.211711+0000", 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020184700000000001 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.900 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:7191 3.4 3:249f6365:::rbd_data.112d64676e7f.0000000000000008:8 [read 163840~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:13.208632+0000", 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.5697906509999999, 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037631119999999999, 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 7191 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208632+0000", 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208632+0000", 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208633+0000", 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5099999999999998e-07 2024-09-16T13:17:16.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208638+0000", 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0599999999999998e-06 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208651+0000", 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2480000000000001e-05 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.208670+0000", 2024-09-16T13:17:16.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9021e-05 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.212214+0000", 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035438969999999998 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.212288+0000", 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4683000000000003e-05 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:16.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:16.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:13.212395+0000", 2024-09-16T13:17:16.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010702 2024-09-16T13:17:16.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.905 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:16.905 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:16.905 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:16.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:16.907 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:16.900+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:17.007 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:17.000+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:17.012 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:17.013 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:17.013 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:17.013 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:17.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:17.108 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:17.104+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:17.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:17:17.174 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:17.174 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:17.174 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:17.174 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:17.174 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:17.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:17.209 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:17.202+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:17.241 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone7 from 0xbfa228d (overlap 0x1bfed4c) to 0x193225e 2024-09-16T13:17:17.286 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:17.286 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:17.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:17.286 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1264 3.4 e27/23 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head v 27'4, mlcod=26'2)", 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 17.778928376, 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026410209999999999, 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501250+0000", 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2410000000000001e-06 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501259+0000", 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501277+0000", 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8683999999999999e-05 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501288+0000", 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.065e-05 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501562+0000", 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027421200000000002 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501618+0000", 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6045000000000003e-05 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503841+0000", 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022230829999999998 2024-09-16T13:17:17.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503890+0000", 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8526000000000003e-05 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.272784688, 2024-09-16T13:17:17.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:17:17.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.295 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.271999189000001, 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:17:17.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:17:17.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.270832266999999, 2024-09-16T13:17:17.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.267957636, 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:17:17.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:17:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:17:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:17:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:17:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:17:17.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.264943896, 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:17:17.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.314 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.239739693000001, 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:17:17.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:17:17.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.473590804000001, 2024-09-16T13:17:17.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:17:17.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5115 3.4 e34/23 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head v 34'14, mlcod=31'12)", 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.4885426979999998, 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029400680000000001, 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791635+0000", 2024-09-16T13:17:17.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4200000000000005e-07 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791643+0000", 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.157e-06 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791660+0000", 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6246e-05 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791666+0000", 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791743+0000", 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7000000000000001e-05 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791756+0000", 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2729e-05 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794523+0000", 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027675949999999999 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794575+0000", 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1082000000000003e-05 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.328 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5290 3.9 e34/23 3:975e9c33:::rbd_data.112d64676e7f.00000000000000b8:head v 34'26, mlcod=32'24)", 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.958105+0000", 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.3220722800000004, 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002485054, 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958105+0000", 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958105+0000", 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958106+0000", 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2139999999999999e-06 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958117+0000", 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1188e-05 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958143+0000", 2024-09-16T13:17:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5670999999999999e-05 2024-09-16T13:17:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958152+0000", 2024-09-16T13:17:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8389999999999996e-06 2024-09-16T13:17:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958204+0000", 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2219999999999998e-05 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958226+0000", 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2334e-05 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960538+0000", 2024-09-16T13:17:17.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023112779999999999 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960590+0000", 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2309999999999999e-05 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5435 3.4 e34/23 3:248e1f52:::rbd_data.112d64676e7f.0000000000000036:head v 34'21, mlcod=34'19)", 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:08.157555+0000", 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.1226221340000002, 2024-09-16T13:17:17.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024120550000000002, 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157555+0000", 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157555+0000", 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157556+0000", 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209999999999999e-06 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157590+0000", 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3918000000000003e-05 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157608+0000", 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8059000000000001e-05 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157615+0000", 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7129999999999999e-06 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157662+0000", 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7250000000000003e-05 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.157678+0000", 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5577999999999999e-05 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.159928+0000", 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002250087 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:08.159967+0000", 2024-09-16T13:17:17.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9329000000000002e-05 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.2214885009999996, 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:17:17.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.2208790609999998, 2024-09-16T13:17:17.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:17:17.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:17:17.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6815 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'44, mlcod=36'42)", 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.149529+0000", 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.1306479229999997, 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024510030000000002, 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149529+0000", 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149529+0000", 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149531+0000", 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0530000000000001e-06 2024-09-16T13:17:17.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149642+0000", 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000110951 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149685+0000", 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3075999999999999e-05 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149704+0000", 2024-09-16T13:17:17.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8592000000000001e-05 2024-09-16T13:17:17.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149818+0000", 2024-09-16T13:17:17.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113842 2024-09-16T13:17:17.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.149876+0000", 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8522e-05 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.151895+0000", 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020183179999999998 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.151980+0000", 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5649000000000006e-05 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.351 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.352 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.352 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6864 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'46, mlcod=37'44)", 2024-09-16T13:17:17.352 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.300990+0000", 2024-09-16T13:17:17.352 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.9791870060000001, 2024-09-16T13:17:17.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022853959999999999, 2024-09-16T13:17:17.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.352 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.300990+0000", 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.300990+0000", 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.300992+0000", 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0439999999999998e-06 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.301014+0000", 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2166000000000001e-05 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.301051+0000", 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6891999999999998e-05 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.301064+0000", 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.291e-05 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.301188+0000", 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012352000000000001 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.301273+0000", 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5600999999999996e-05 2024-09-16T13:17:17.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.303145+0000", 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018716760000000001 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.303276+0000", 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013058699999999999 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6963 3.4 e37/23 3:2713f365:::rbd_data.112d6fb8d387.0000000000000133:head v 37'31, mlcod=37'29)", 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.7243149110000004, 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027598190000000002, 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:17.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555864+0000", 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555882+0000", 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7816e-05 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555909+0000", 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7838999999999999e-05 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555923+0000", 2024-09-16T13:17:17.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3577000000000001e-05 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556083+0000", 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015987299999999999 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556132+0000", 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8655000000000001e-05 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558539+0000", 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024076969999999999 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558622+0000", 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2777999999999999e-05 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.7235952470000004, 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:17:17.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.7224944310000003, 2024-09-16T13:17:17.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:17:17.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.7223547909999999, 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:17:17.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:17:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.7218542609999998, 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:17:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.380 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:17.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:17.382 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #6, image image_client.0-clone6 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent7 2024-09-16T13:17:17.382 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:17.374+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:17.383 INFO:teuthology.orchestra.run.smithi098.stdout:1405 write 0x59ad191 thru 0x59b084e (0x36be bytes) 2024-09-16T13:17:17.399 INFO:teuthology.orchestra.run.smithi098.stdout:1407 read 0x3c07be3 thru 0x3c164f5 (0xe913 bytes) 2024-09-16T13:17:17.418 INFO:teuthology.orchestra.run.smithi098.stdout:1409 write 0xebb38d9 thru 0xebbe148 (0xa870 bytes) 2024-09-16T13:17:17.424 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:17.424 INFO:teuthology.orchestra.run.smithi098.stdout:1411 write 0xba46287 thru 0xba4b2de (0x5058 bytes) 2024-09-16T13:17:17.430 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:17.430 INFO:teuthology.orchestra.run.smithi098.stdout:1414 read 0x2768b80 thru 0x276e532 (0x59b3 bytes) 2024-09-16T13:17:17.430 INFO:teuthology.orchestra.run.smithi098.stdout:1415 punch from 0xe440951 to 0xe44103f, (0x6ee bytes) 2024-09-16T13:17:17.430 INFO:teuthology.orchestra.run.smithi098.stdout:1416 writesame 0x208487 thru 0x20d320 data_size 0x1(0x4e9a bytes) 2024-09-16T13:17:17.481 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:17.476+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:17.486 INFO:teuthology.orchestra.run.smithi098.stdout:1417 flatten 2024-09-16T13:17:17.506 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:17.506 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:17.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.506 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11540 3.8 3.dd9a4208 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.506 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497186+0000", 2024-09-16T13:17:17.506 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.001028922, 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001057138, 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11540 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497186+0000", 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497186+0000", 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497187+0000", 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.46e-07 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497189+0000", 2024-09-16T13:17:17.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8619999999999999e-06 2024-09-16T13:17:17.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497194+0000", 2024-09-16T13:17:17.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7860000000000002e-06 2024-09-16T13:17:17.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497198+0000", 2024-09-16T13:17:17.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.641e-06 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11569 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497202+0000", 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.001013239, 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00110581, 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.510 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11569 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497202+0000", 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497202+0000", 2024-09-16T13:17:17.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497204+0000", 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9209999999999999e-06 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497206+0000", 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9759999999999998e-06 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497212+0000", 2024-09-16T13:17:17.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0669999999999997e-06 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497217+0000", 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7840000000000003e-06 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11571 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497221+0000", 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00099401200000000006, 2024-09-16T13:17:17.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001143006, 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11571 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497221+0000", 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497221+0000", 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497222+0000", 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5599999999999995e-07 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497224+0000", 2024-09-16T13:17:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9759999999999998e-06 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497230+0000", 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9190000000000004e-06 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497233+0000", 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7469999999999999e-06 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11572 3.18 3.7c4cbb18 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497238+0000", 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00097763699999999995, 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011840570000000001, 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11572 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497238+0000", 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497238+0000", 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497238+0000", 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1399999999999997e-07 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497240+0000", 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.838e-06 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497246+0000", 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1449999999999996e-06 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497250+0000", 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6090000000000001e-06 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11573 3.8 3.dd9a4208 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.520 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497254+0000", 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.000961279, 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0012236479999999999, 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11573 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497254+0000", 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497254+0000", 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497254+0000", 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:17:17.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497256+0000", 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.871e-06 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497262+0000", 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1859999999999997e-06 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497266+0000", 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3510000000000002e-06 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11574 3.18 3.7c4cbb18 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497270+0000", 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.000945421, 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001265831, 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11574 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497270+0000", 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497270+0000", 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497270+0000", 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4200000000000002e-07 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497276+0000", 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3489999999999999e-06 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497282+0000", 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9669999999999998e-06 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497285+0000", 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.568e-06 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11361 3.7 3:e6a67092:::rbd_data.112de4fe95f7.0000000000000012:a [read 94208~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496424+0000", 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0017916239999999999, 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021718010000000001, 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11361 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496424+0000", 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496424+0000", 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496424+0000", 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.08e-07 2024-09-16T13:17:17.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496426+0000", 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.956e-06 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496432+0000", 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.088e-06 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496436+0000", 2024-09-16T13:17:17.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8979999999999998e-06 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.498436+0000", 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020003400000000002 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.498540+0000", 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000104132 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11575 3.8 3.dd9a4208 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497289+0000", 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00092631799999999998, 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013706969999999999, 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11575 2024-09-16T13:17:17.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497289+0000", 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497289+0000", 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497289+0000", 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5999999999999999e-07 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497291+0000", 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7740000000000001e-06 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497297+0000", 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7570000000000001e-06 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497301+0000", 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.583e-06 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11362 3.7 3:e6a67092:::rbd_data.112de4fe95f7.0000000000000012:a [read 167936~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.535 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496440+0000", 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0017750929999999999, 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022816669999999998, 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11362 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496440+0000", 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496440+0000", 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496441+0000", 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3600000000000004e-07 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496443+0000", 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.34e-06 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496449+0000", 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.207e-06 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496453+0000", 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.873e-06 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.498699+0000", 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022456260000000001 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11579 3.7 3.e918a767 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497305+0000", 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00090992199999999997, 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014869880000000001, 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.539 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11579 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497305+0000", 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497305+0000", 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497306+0000", 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497307+0000", 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7770000000000001e-06 2024-09-16T13:17:17.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497313+0000", 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.947e-06 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497317+0000", 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3579999999999999e-06 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.542 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11581 3.7 3.e918a767 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497321+0000", 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00089464799999999995, 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015259080000000001, 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11581 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497321+0000", 2024-09-16T13:17:17.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497321+0000", 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497321+0000", 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7e-07 2024-09-16T13:17:17.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497323+0000", 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.716e-06 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497329+0000", 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0059999999999998e-06 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497332+0000", 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.506e-06 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11382 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496473+0000", 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0017418830000000001, 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00242677, 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.546 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11382 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496473+0000", 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496473+0000", 2024-09-16T13:17:17.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496474+0000", 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.51e-07 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496476+0000", 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0559999999999999e-06 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496482+0000", 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1920000000000002e-06 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496486+0000", 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6109999999999998e-06 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.549 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11596 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497336+0000", 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00087897699999999997, 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0016177419999999999, 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11596 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497336+0000", 2024-09-16T13:17:17.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497336+0000", 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497337+0000", 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.58e-07 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497339+0000", 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497345+0000", 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.348e-06 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497349+0000", 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5899999999999999e-06 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.552 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11383 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496490+0000", 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0017256859999999999, 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002523904, 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.553 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11383 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496490+0000", 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496490+0000", 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496490+0000", 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496492+0000", 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7659999999999999e-06 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496497+0000", 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6869999999999999e-06 2024-09-16T13:17:17.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496505+0000", 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4359999999999996e-06 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11597 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497353+0000", 2024-09-16T13:17:17.556 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00086272200000000001, 2024-09-16T13:17:17.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001718532, 2024-09-16T13:17:17.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.557 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.557 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.557 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.557 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11597 2024-09-16T13:17:17.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.557 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497353+0000", 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497353+0000", 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497353+0000", 2024-09-16T13:17:17.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497355+0000", 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7370000000000001e-06 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497361+0000", 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8680000000000004e-06 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497376+0000", 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5316e-05 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11598 3.8 3.dd9a4208 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497381+0000", 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00083454999999999998, 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001750782, 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.560 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11598 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497381+0000", 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497381+0000", 2024-09-16T13:17:17.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497382+0000", 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6300000000000004e-07 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497384+0000", 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497390+0000", 2024-09-16T13:17:17.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.246e-06 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497393+0000", 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5389999999999999e-06 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11599 3.8 3.dd9a4208 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497398+0000", 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00081742100000000003, 2024-09-16T13:17:17.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017887140000000001, 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11599 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497398+0000", 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497398+0000", 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497398+0000", 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1500000000000005e-07 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497400+0000", 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0679999999999999e-06 2024-09-16T13:17:17.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497406+0000", 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8660000000000002e-06 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497410+0000", 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.636e-06 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11400 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496542+0000", 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0016733919999999999, 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027042110000000002, 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11400 2024-09-16T13:17:17.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496542+0000", 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496542+0000", 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496542+0000", 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6399999999999998e-07 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496548+0000", 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7780000000000004e-06 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496554+0000", 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0270000000000001e-06 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496558+0000", 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6720000000000002e-06 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11647 3.18 3.7c4cbb18 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497414+0000", 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.000801306, 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001886836, 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.570 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11647 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497414+0000", 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497414+0000", 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497414+0000", 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2199999999999999e-07 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497417+0000", 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497423+0000", 2024-09-16T13:17:17.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2419999999999997e-06 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497427+0000", 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8349999999999997e-06 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11402 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496562+0000", 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.001653198, 2024-09-16T13:17:17.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027931060000000001, 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11402 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496562+0000", 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496562+0000", 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496563+0000", 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1300000000000001e-07 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496564+0000", 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.934e-06 2024-09-16T13:17:17.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496570+0000", 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9619999999999998e-06 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496574+0000", 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6689999999999999e-06 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11649 3.7 3.e918a767 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497431+0000", 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00078450899999999997, 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0019766649999999998, 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11649 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497431+0000", 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497431+0000", 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497431+0000", 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3000000000000001e-07 2024-09-16T13:17:17.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497433+0000", 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497439+0000", 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7910000000000001e-06 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497442+0000", 2024-09-16T13:17:17.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.145e-06 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11652 3.18 3.7c4cbb18 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497446+0000", 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00076949600000000005, 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002016384, 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.580 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.581 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11652 2024-09-16T13:17:17.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.581 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497446+0000", 2024-09-16T13:17:17.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497446+0000", 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497446+0000", 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4499999999999997e-07 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497448+0000", 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.824e-06 2024-09-16T13:17:17.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497454+0000", 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9020000000000004e-06 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497457+0000", 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4920000000000002e-06 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11656 3.7 3.e918a767 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497462+0000", 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00075357600000000005, 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020552809999999999, 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11656 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.584 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497462+0000", 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497462+0000", 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497462+0000", 2024-09-16T13:17:17.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497467+0000", 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.454e-06 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497472+0000", 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2700000000000004e-06 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497475+0000", 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4599999999999999e-06 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11665 3.7 3.490e6567 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497479+0000", 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00073629299999999995, 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020922100000000002, 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.587 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11665 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497479+0000", 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497479+0000", 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497479+0000", 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497486+0000", 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6680000000000004e-06 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497492+0000", 2024-09-16T13:17:17.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.387e-06 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497496+0000", 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7579999999999998e-06 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11666 3.8 3.dd9a4208 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497500+0000", 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00071484900000000004, 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0021256769999999999, 2024-09-16T13:17:17.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11666 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497500+0000", 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497500+0000", 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497501+0000", 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1899999999999998e-07 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497504+0000", 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7310000000000002e-06 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497509+0000", 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7289999999999997e-06 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497513+0000", 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4149999999999999e-06 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11474 3.7 3.e918a767 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.593 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496933+0000", 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0012819470000000001, 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002748861, 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11474 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496933+0000", 2024-09-16T13:17:17.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496933+0000", 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496934+0000", 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4700000000000001e-07 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496936+0000", 2024-09-16T13:17:17.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5469999999999999e-06 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496943+0000", 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6139999999999997e-06 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496947+0000", 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7330000000000001e-06 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11668 3.7 3.490e6567 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497517+0000", 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00069855000000000004, 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002222379, 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11668 2024-09-16T13:17:17.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497517+0000", 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497517+0000", 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497517+0000", 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497519+0000", 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.888e-06 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497525+0000", 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497534+0000", 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3549999999999994e-06 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11475 3.7 3.e918a767 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496951+0000", 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.001264048, 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028425, 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.600 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11475 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496951+0000", 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496951+0000", 2024-09-16T13:17:17.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496952+0000", 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3599999999999999e-07 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496954+0000", 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9e-06 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496960+0000", 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1659999999999999e-06 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496964+0000", 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7969999999999998e-06 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11670 3.8 3.dd9a4208 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497539+0000", 2024-09-16T13:17:17.603 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00067649799999999999, 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002309616, 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11670 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497539+0000", 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497539+0000", 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497539+0000", 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7000000000000004e-07 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497542+0000", 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.086e-06 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497547+0000", 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7529999999999998e-06 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497551+0000", 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6430000000000001e-06 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11687 3.18 3.7c4cbb18 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497555+0000", 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00066061600000000005, 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023465249999999999, 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11687 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497555+0000", 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497555+0000", 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497555+0000", 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:17:17.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497557+0000", 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.863e-06 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497563+0000", 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0630000000000002e-06 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497567+0000", 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-06 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11690 3.7 3.e918a767 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497571+0000", 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00064465300000000002, 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023859620000000002, 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.610 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11690 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497571+0000", 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497571+0000", 2024-09-16T13:17:17.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497571+0000", 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497573+0000", 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497579+0000", 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7740000000000001e-06 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497582+0000", 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.388e-06 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11502 3.7 3.e918a767 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.613 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497000+0000", 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0012149629999999999, 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003010937, 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11502 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497000+0000", 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497000+0000", 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497001+0000", 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497003+0000", 2024-09-16T13:17:17.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9539999999999998e-06 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497009+0000", 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9320000000000001e-06 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497012+0000", 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.732e-06 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11703 3.7 3.490e6567 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497586+0000", 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00062947600000000002, 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024805420000000001, 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11703 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.617 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497586+0000", 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497586+0000", 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497586+0000", 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497596+0000", 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7170000000000003e-06 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497602+0000", 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8109999999999999e-06 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497605+0000", 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1949999999999999e-06 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11503 3.7 3.e918a767 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497038+0000", 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0011777350000000001, 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030833179999999998, 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11503 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497038+0000", 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497038+0000", 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497038+0000", 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5700000000000002e-07 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497040+0000", 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1019999999999999e-06 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497047+0000", 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3770000000000001e-06 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497051+0000", 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7299999999999996e-06 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.623 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11704 3.8 3.dd9a4208 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497609+0000", 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.00060622300000000003, 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002564687, 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11704 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497609+0000", 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497609+0000", 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497609+0000", 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497611+0000", 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8390000000000001e-06 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497617+0000", 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7039999999999999e-06 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497620+0000", 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4690000000000002e-06 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.626 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11516 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497056+0000", 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0011596899999999999, 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031817759999999999, 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11516 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.627 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497056+0000", 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497056+0000", 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497056+0000", 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0399999999999996e-07 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497062+0000", 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3180000000000003e-06 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497069+0000", 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2090000000000002e-06 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497072+0000", 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3100000000000001e-06 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11517 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497076+0000", 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.001139461, 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032180479999999998, 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.630 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11517 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497076+0000", 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497076+0000", 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497076+0000", 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497078+0000", 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8160000000000001e-06 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497084+0000", 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0070000000000003e-06 2024-09-16T13:17:17.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497088+0000", 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6710000000000001e-06 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11755 3.8 3.c2081508 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499572+0000", 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4294967295.9986434, 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00077787500000000003, 2024-09-16T13:17:17.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11755 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499572+0000", 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499572+0000", 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499572+0000", 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0100000000000005e-07 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499575+0000", 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6699999999999998e-06 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499583+0000", 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9990000000000008e-06 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499587+0000", 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.42e-06 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11537 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497132+0000", 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.001083215, 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032726669999999999, 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11537 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497132+0000", 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497132+0000", 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497133+0000", 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3599999999999999e-07 2024-09-16T13:17:17.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497134+0000", 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8479999999999999e-06 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497140+0000", 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.896e-06 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497144+0000", 2024-09-16T13:17:17.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3450000000000002e-06 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11538 3.7 3.490e6567 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497148+0000", 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.001067709, 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003315456, 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.640 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11538 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497148+0000", 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497148+0000", 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497148+0000", 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7599999999999998e-07 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497150+0000", 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7349999999999999e-06 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497155+0000", 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8000000000000004e-06 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.643 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497160+0000", 2024-09-16T13:17:17.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7360000000000001e-06 2024-09-16T13:17:17.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.643 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.643 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:17.643 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 40 2024-09-16T13:17:17.643 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:17.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:17.646 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:17.646 INFO:teuthology.orchestra.run.smithi098.stdout:1420 trunc from 0xebbe149 to 0xa514ac0 2024-09-16T13:17:17.646 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:17.646 INFO:teuthology.orchestra.run.smithi098.stdout:1423 read 0x1dd7324 thru 0x1de5ab0 (0xe78d bytes) 2024-09-16T13:17:17.647 INFO:teuthology.orchestra.run.smithi098.stdout:1424 write 0x791d5f7 thru 0x79258f9 (0x8303 bytes) 2024-09-16T13:17:17.647 INFO:teuthology.orchestra.run.smithi098.stdout:1427 writesame 0x4353965 thru 0x435d103 data_size 0x7(0x979f bytes) 2024-09-16T13:17:17.647 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:17.647 INFO:teuthology.orchestra.run.smithi098.stdout:1430 read 0x6592577 thru 0x65a1bfd (0xf687 bytes) 2024-09-16T13:17:17.647 INFO:teuthology.orchestra.run.smithi098.stdout:1431 write 0x45baae2 thru 0x45c2b7d (0x809c bytes) 2024-09-16T13:17:17.647 INFO:teuthology.orchestra.run.smithi098.stdout:1433 write 0x3c3bb6e thru 0x3c3ccbf (0x1152 bytes) 2024-09-16T13:17:17.647 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:17.638+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:17.667 INFO:teuthology.orchestra.run.smithi098.stdout:1434 write 0xc789a29 thru 0xc792b61 (0x9139 bytes) 2024-09-16T13:17:17.674 INFO:teuthology.orchestra.run.smithi098.stdout:1435 read 0x9f2caf2 thru 0x9f3a653 (0xdb62 bytes) 2024-09-16T13:17:17.675 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:17.675 INFO:teuthology.orchestra.run.smithi098.stdout:1440 write 0xc4ce1a8 thru 0xc4d2229 (0x4082 bytes) 2024-09-16T13:17:17.683 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:17.683 INFO:teuthology.orchestra.run.smithi098.stdout:1443 read 0x8100f54 thru 0x8102665 (0x1712 bytes) 2024-09-16T13:17:17.683 INFO:teuthology.orchestra.run.smithi098.stdout:1444 read 0xcd9557 thru 0xcdc714 (0x31be bytes) 2024-09-16T13:17:17.684 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:17.684 INFO:teuthology.orchestra.run.smithi098.stdout:1447 writesame 0x364840a thru 0x364d7e9 data_size 0x2c(0x53e0 bytes) 2024-09-16T13:17:17.688 INFO:teuthology.orchestra.run.smithi098.stdout:1449 read 0x805036f thru 0x80565dc (0x626e bytes) 2024-09-16T13:17:17.689 INFO:teuthology.orchestra.run.smithi098.stdout:1450 writesame 0x1d9d2e8 thru 0x1daaa77 data_size 0x8(0xd790 bytes) 2024-09-16T13:17:17.722 INFO:teuthology.orchestra.run.smithi098.stdout:1452 trunc from 0xc792b62 to 0x33b2109 2024-09-16T13:17:17.746 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:17.738+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:17.747 INFO:teuthology.orchestra.run.smithi098.stdout:1453 trunc from 0x33b2109 to 0x28b0936 2024-09-16T13:17:17.775 INFO:teuthology.orchestra.run.smithi098.stdout:1454 trunc from 0x28b0936 to 0x866d20f 2024-09-16T13:17:17.778 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:17.778 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:17.779 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:17.779 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:17.787 INFO:teuthology.orchestra.run.smithi098.stdout:1455 read 0x7c45a76 thru 0x7c52c92 (0xd21d bytes) 2024-09-16T13:17:17.788 INFO:teuthology.orchestra.run.smithi098.stdout:1456 writesame 0x81513e0 thru 0x815fdfa data_size 0x3(0xea1b bytes) 2024-09-16T13:17:17.788 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:17.806 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:17.831 INFO:teuthology.orchestra.run.smithi098.stdout:1459 write 0xd07162d thru 0xd0811e2 (0xfbb6 bytes) 2024-09-16T13:17:17.838 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:17.838 INFO:teuthology.orchestra.run.smithi098.stdout:1461 read 0x932f7de thru 0x93322d7 (0x2afa bytes) 2024-09-16T13:17:17.838 INFO:teuthology.orchestra.run.smithi098.stdout:1463 writesame 0xaaee8d7 thru 0xaaf7aff data_size 0x1f(0x9229 bytes) 2024-09-16T13:17:17.847 INFO:teuthology.orchestra.run.smithi098.stdout:1464 read 0x38c2ed9 thru 0x38c6a40 (0x3b68 bytes) 2024-09-16T13:17:17.847 INFO:teuthology.orchestra.run.smithi098.stdout:1465 punch from 0x5abe575 to 0x5abe91e, (0x3a9 bytes) 2024-09-16T13:17:17.847 INFO:teuthology.orchestra.run.smithi098.stdout:1467 punch from 0xd3ab78 to 0xd4a371, (0xf7f9 bytes) 2024-09-16T13:17:17.847 INFO:teuthology.orchestra.run.smithi098.stdout:1469 write 0xc716754 thru 0xc721823 (0xb0d0 bytes) 2024-09-16T13:17:17.847 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:17.840+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:17.853 INFO:teuthology.orchestra.run.smithi098.stdout:1471 write 0x39eee0f thru 0x39fe371 (0xf563 bytes) 2024-09-16T13:17:17.878 INFO:teuthology.orchestra.run.smithi098.stdout:1472 trunc from 0xd0811e3 to 0x2da9e03 2024-09-16T13:17:17.905 INFO:teuthology.orchestra.run.smithi098.stdout:1473 read 0x2d0ff91 thru 0x2d1322d (0x329d bytes) 2024-09-16T13:17:17.906 INFO:teuthology.orchestra.run.smithi098.stdout:1474 punch from 0x2c796f7 to 0x2c80440, (0x6d49 bytes) 2024-09-16T13:17:17.906 INFO:teuthology.orchestra.run.smithi098.stdout:1475 read 0x19f0bc5 thru 0x19fd19e (0xc5da bytes) 2024-09-16T13:17:17.907 INFO:teuthology.orchestra.run.smithi098.stdout:1477 read 0x1df0410 thru 0x1dfbb76 (0xb767 bytes) 2024-09-16T13:17:17.915 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:17.915 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:17.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:17.915 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:17.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.915 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5101 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.915 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:17.916 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.128899618, 2024-09-16T13:17:17.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595248, 2024-09-16T13:17:17.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.916 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.916 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.916 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.916 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.917 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5101 2024-09-16T13:17:17.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.917 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:17.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:17.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:17.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:17.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777333+0000", 2024-09-16T13:17:17.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1949999999999996e-06 2024-09-16T13:17:17.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777342+0000", 2024-09-16T13:17:17.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6950000000000006e-06 2024-09-16T13:17:17.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777348+0000", 2024-09-16T13:17:17.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:17:17.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785241+0000", 2024-09-16T13:17:17.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078929329999999995 2024-09-16T13:17:17.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785391+0000", 2024-09-16T13:17:17.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150113 2024-09-16T13:17:17.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785969+0000", 2024-09-16T13:17:17.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057822899999999998 2024-09-16T13:17:17.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789745+0000", 2024-09-16T13:17:17.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037756529999999999 2024-09-16T13:17:17.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793823+0000", 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040786299999999998 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793880+0000", 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7157999999999997e-05 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793924+0000", 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3407e-05 2024-09-16T13:17:17.924 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.925 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.925 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.925 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5102 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.925 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:17.925 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.128852326000001, 2024-09-16T13:17:17.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016663710000000002, 2024-09-16T13:17:17.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5102 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777377+0000", 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4600000000000003e-07 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777381+0000", 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5290000000000002e-06 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777393+0000", 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2001000000000001e-05 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777401+0000", 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3869999999999997e-06 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786753+0000", 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093523649999999996 2024-09-16T13:17:17.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786860+0000", 2024-09-16T13:17:17.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106921 2024-09-16T13:17:17.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787286+0000", 2024-09-16T13:17:17.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042651800000000003 2024-09-16T13:17:17.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793815+0000", 2024-09-16T13:17:17.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065290579999999999 2024-09-16T13:17:17.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793932+0000", 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116688 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794002+0000", 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0313000000000003e-05 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794040+0000", 2024-09-16T13:17:17.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7083999999999999e-05 2024-09-16T13:17:17.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.932 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.932 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5103 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.932 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:17.932 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.128791440000001, 2024-09-16T13:17:17.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017505544000000001, 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5103 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:17.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777438+0000", 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.78e-07 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777442+0000", 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5950000000000001e-06 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777453+0000", 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0372e-05 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777458+0000", 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6559999999999997e-06 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788034+0000", 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010575753 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788111+0000", 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7038000000000003e-05 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788463+0000", 2024-09-16T13:17:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000352028 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794263+0000", 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005800342 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794837+0000", 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057334300000000003 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794897+0000", 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0690000000000003e-05 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794942+0000", 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4848999999999997e-05 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5104 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.128747032, 2024-09-16T13:17:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017694142999999999, 2024-09-16T13:17:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5104 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:17.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777482+0000", 2024-09-16T13:17:17.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:17:17.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777487+0000", 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6639999999999997e-06 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777496+0000", 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2890000000000003e-06 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777501+0000", 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2730000000000002e-06 2024-09-16T13:17:17.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788951+0000", 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011449429000000001 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789007+0000", 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5643999999999998e-05 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789205+0000", 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019820800000000001 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794268+0000", 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050636850000000001 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795094+0000", 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082569700000000002 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795139+0000", 2024-09-16T13:17:17.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5055000000000001e-05 2024-09-16T13:17:17.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795175+0000", 2024-09-16T13:17:17.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6149000000000002e-05 2024-09-16T13:17:17.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.950 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5105 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.950 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:17.950 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.128684691, 2024-09-16T13:17:17.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017729482000000001, 2024-09-16T13:17:17.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5105 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777544+0000", 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1100000000000004e-07 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777549+0000", 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:17.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777561+0000", 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1524e-05 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777566+0000", 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.558e-06 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789781+0000", 2024-09-16T13:17:17.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012214981 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789870+0000", 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9171000000000004e-05 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790313+0000", 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044255299999999999 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794273+0000", 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039596129999999998 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795213+0000", 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094086099999999998 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795239+0000", 2024-09-16T13:17:17.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5324e-05 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795273+0000", 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4198e-05 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.956 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5106 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.128639547000001, 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018621867, 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5106 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777590+0000", 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:17.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777594+0000", 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6750000000000001e-06 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777604+0000", 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0076e-05 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777610+0000", 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7529999999999998e-06 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790893+0000", 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013282845 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790999+0000", 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106289 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791493+0000", 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049357800000000005 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794662+0000", 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031691409999999999 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796146+0000", 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00148375 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796176+0000", 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0349e-05 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796211+0000", 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4530000000000003e-05 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.962 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5107 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.128608286, 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018745067000000001, 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5107 2024-09-16T13:17:17.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777621+0000", 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-07 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777624+0000", 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7950000000000001e-06 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777633+0000", 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5979999999999997e-06 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777638+0000", 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.989e-06 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792024+0000", 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014385692 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792104+0000", 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0567000000000003e-05 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792340+0000", 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023624200000000001 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794667+0000", 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023269990000000002 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796304+0000", 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001636981 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796331+0000", 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6492999999999999e-05 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796365+0000", 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4102e-05 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5108 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.128554457, 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019090617000000001, 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5108 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777677+0000", 2024-09-16T13:17:17.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7879999999999998e-06 2024-09-16T13:17:17.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777686+0000", 2024-09-16T13:17:17.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7679999999999997e-06 2024-09-16T13:17:17.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777689+0000", 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.123e-06 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792751+0000", 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01506158 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792855+0000", 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010456 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793305+0000", 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045006100000000003 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794671+0000", 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001366304 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796393+0000", 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001721272 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796733+0000", 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000340611 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796764+0000", 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.099e-05 2024-09-16T13:17:17.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5208 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.026795006, 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014882177999999999, 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5208 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879434+0000", 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.322e-06 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879443+0000", 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879459+0000", 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6133999999999998e-05 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879467+0000", 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2400000000000007e-06 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888339+0000", 2024-09-16T13:17:17.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088721619999999994 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888461+0000", 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121341 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888819+0000", 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035782799999999998 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891271+0000", 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002452872 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894224+0000", 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029522910000000001 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894275+0000", 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1647999999999997e-05 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894315+0000", 2024-09-16T13:17:17.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9993999999999997e-05 2024-09-16T13:17:17.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.981 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5210 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.026239818000001, 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01466423, 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5210 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879989+0000", 2024-09-16T13:17:17.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8800000000000001e-07 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879995+0000", 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880017+0000", 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2101000000000001e-05 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880024+0000", 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9240000000000001e-06 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890250+0000", 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010226012 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890344+0000", 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.446e-05 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890770+0000", 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042575099999999998 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894344+0000", 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003574163 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894549+0000", 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020477599999999999 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894606+0000", 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6891000000000001e-05 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894653+0000", 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6867999999999999e-05 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5211 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.026195755, 2024-09-16T13:17:17.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014800604, 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5211 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880033+0000", 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.06e-07 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880039+0000", 2024-09-16T13:17:17.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880052+0000", 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-05 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880059+0000", 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3090000000000001e-06 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891316+0000", 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011257118999999999 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891374+0000", 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8020999999999999e-05 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891577+0000", 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020371699999999999 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894348+0000", 2024-09-16T13:17:17.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027708300000000002 2024-09-16T13:17:17.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894726+0000", 2024-09-16T13:17:17.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037819399999999999 2024-09-16T13:17:17.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894784+0000", 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7667999999999997e-05 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894833+0000", 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8936e-05 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5212 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.993 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.025840381, 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014645704000000001, 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5212 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880389+0000", 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1680000000000001e-06 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880395+0000", 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:17:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:17.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880408+0000", 2024-09-16T13:17:17.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.308e-05 2024-09-16T13:17:17.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:17.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880414+0000", 2024-09-16T13:17:17.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.105e-06 2024-09-16T13:17:17.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892037+0000", 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011622402 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892116+0000", 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9939999999999997e-05 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892597+0000", 2024-09-16T13:17:17.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048076500000000002 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894492+0000", 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001895246 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894921+0000", 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000428114 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894985+0000", 2024-09-16T13:17:17.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4777999999999995e-05 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895034+0000", 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8133999999999997e-05 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5213 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.025707231, 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014632608, 2024-09-16T13:17:17.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5213 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880522+0000", 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2600000000000001e-07 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880527+0000", 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:18.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880538+0000", 2024-09-16T13:17:18.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917999999999999e-05 2024-09-16T13:17:18.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880543+0000", 2024-09-16T13:17:18.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.361e-06 2024-09-16T13:17:18.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893069+0000", 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012525497999999999 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893152+0000", 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2805000000000004e-05 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893635+0000", 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048315100000000002 2024-09-16T13:17:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:18.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894852+0000", 2024-09-16T13:17:18.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001216806 2024-09-16T13:17:18.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895075+0000", 2024-09-16T13:17:18.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022300299999999999 2024-09-16T13:17:18.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895122+0000", 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7077999999999999e-05 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895154+0000", 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1919999999999999e-05 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5214 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.025552579999999, 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017785977000000001, 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5214 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:18.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880681+0000", 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880689+0000", 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3610000000000001e-06 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880694+0000", 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5809999999999998e-06 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895186+0000", 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014492430000000001 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895311+0000", 2024-09-16T13:17:18.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012518300000000001 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895760+0000", 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044864900000000002 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898337+0000", 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025765229999999998 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898354+0000", 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7538999999999999e-05 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898406+0000", 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2054e-05 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898462+0000", 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5404e-05 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.011 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5215 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.025287746, 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018191319000000001, 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:18.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.013 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5215 2024-09-16T13:17:18.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.013 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:18.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:18.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:18.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880941+0000", 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880946+0000", 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3710000000000002e-06 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880955+0000", 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1230000000000006e-06 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880959+0000", 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896782+0000", 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015822716000000001 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896876+0000", 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3667999999999998e-05 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.897289+0000", 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041357399999999999 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898474+0000", 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011847209999999999 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899036+0000", 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056215699999999998 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:18.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899090+0000", 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3662999999999998e-05 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899132+0000", 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1814000000000003e-05 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5282 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:18.018 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.9566156320000001, 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014924988, 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5282 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:18.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.53e-07 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949635+0000", 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2422999999999999e-05 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949641+0000", 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.896e-06 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949644+0000", 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5270000000000001e-06 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959656+0000", 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010012378000000001 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959747+0000", 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1273000000000005e-05 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960045+0000", 2024-09-16T13:17:18.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029733 2024-09-16T13:17:18.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:18.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963196+0000", 2024-09-16T13:17:18.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031512620000000002 2024-09-16T13:17:18.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964453+0000", 2024-09-16T13:17:18.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001257357 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964502+0000", 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9044e-05 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964538+0000", 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5244999999999999e-05 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.024 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5283 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.9565804779999993, 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015065762, 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:18.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5283 2024-09-16T13:17:18.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.026 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:18.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:18.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:18.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949650+0000", 2024-09-16T13:17:18.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.736e-06 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949653+0000", 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2969999999999999e-06 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949655+0000", 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1890000000000001e-06 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:18.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960643+0000", 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010988015 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960732+0000", 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9097000000000004e-05 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961195+0000", 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000462292 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963208+0000", 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002013641 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964585+0000", 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013765769999999999 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964669+0000", 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3863000000000007e-05 2024-09-16T13:17:18.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964713+0000", 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4706999999999997e-05 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5284 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:18.031 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.9565706089999999, 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015152802, 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5284 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:18.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5899999999999998e-07 2024-09-16T13:17:18.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:18.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949659+0000", 2024-09-16T13:17:18.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.601e-06 2024-09-16T13:17:18.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949672+0000", 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2364999999999999e-05 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949674+0000", 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:18.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961869+0000", 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012194903999999999 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961971+0000", 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102154 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.962502+0000", 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00053037500000000003 2024-09-16T13:17:18.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964751+0000", 2024-09-16T13:17:18.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022487800000000001 2024-09-16T13:17:18.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:18.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964755+0000", 2024-09-16T13:17:18.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:18.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964789+0000", 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4054000000000003e-05 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964810+0000", 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0922000000000001e-05 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.038 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5285 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.9565516909999996, 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015553022, 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5285 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949677+0000", 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6899999999999999e-07 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:18.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949679+0000", 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0090000000000002e-06 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949682+0000", 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2550000000000002e-06 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949685+0000", 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.621e-06 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:18.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963409+0000", 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013724485999999999 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963497+0000", 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7560999999999997e-05 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963933+0000", 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043592700000000002 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964820+0000", 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088772299999999998 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965165+0000", 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000344426 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965201+0000", 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5967000000000003e-05 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965230+0000", 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11223 3.8 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 33927~20122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.44537936099999997, 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019333487, 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11223 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460851+0000", 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5730000000000002e-06 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464824+0000", 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039725730000000001 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464861+0000", 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7233000000000001e-05 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464872+0000", 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209000000000001e-05 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464987+0000", 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011423 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465188+0000", 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020091600000000001 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465929+0000", 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00074102 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.475697+0000", 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009768367 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480068+0000", 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043713290000000002 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480143+0000", 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5000999999999996e-05 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480182+0000", 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9035999999999997e-05 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:18.051 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:18.053 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:18.046+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:18.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:1478 write 0x84f836d thru 0x84fa510 (0x21a4 bytes) 2024-09-16T13:17:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:1479 read 0x2daa36b thru 0x2dba0ce (0xfd64 bytes) 2024-09-16T13:17:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:1480 writesame 0xbb7edf5 thru 0xbb8b05f data_size 0x1(0xc26b bytes) 2024-09-16T13:17:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:1481 read 0xbb09f1b thru 0xbb174c2 (0xd5a8 bytes) 2024-09-16T13:17:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:1482 punch from 0x3ff06d5 to 0x3ff3abc, (0x33e7 bytes) 2024-09-16T13:17:18.057 INFO:teuthology.orchestra.run.smithi098.stdout:1483 read 0x456cdde thru 0x457a2af (0xd4d2 bytes) 2024-09-16T13:17:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:1484 read 0x4d68385 thru 0x4d6b84d (0x34c9 bytes) 2024-09-16T13:17:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:1485 trunc from 0xbb8b060 to 0xa740dd6 2024-09-16T13:17:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:1486 writesame 0x98ee502 thru 0x98f2e91 data_size 0x2(0x4990 bytes) 2024-09-16T13:17:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:1487 read 0x4324f39 thru 0x4334b3d (0xfc05 bytes) 2024-09-16T13:17:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:1489 trunc from 0xa740dd6 to 0x4dd9d8e 2024-09-16T13:17:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:1490 punch from 0x23cbc24 to 0x23d5e5e, (0xa23a bytes) 2024-09-16T13:17:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.058 INFO:teuthology.orchestra.run.smithi098.stdout:1492 write 0x37cb6db thru 0x37d70f4 (0xba1a bytes) 2024-09-16T13:17:18.059 INFO:teuthology.orchestra.run.smithi098.stdout:1493 punch from 0x20de0b3 to 0x20ea673, (0xc5c0 bytes) 2024-09-16T13:17:18.059 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.059 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.059 INFO:teuthology.orchestra.run.smithi098.stdout:1496 read 0x83b14e thru 0x841401 (0x62b4 bytes) 2024-09-16T13:17:18.060 INFO:teuthology.orchestra.run.smithi098.stdout:1497 trunc from 0x4dd9d8e to 0xeb17392 2024-09-16T13:17:18.073 INFO:teuthology.orchestra.run.smithi098.stdout:1498 read 0xe0e68c9 thru 0xe0eafd4 (0x470c bytes) 2024-09-16T13:17:18.091 INFO:teuthology.orchestra.run.smithi098.stdout:1506 trunc from 0xeb17392 to 0x59d42c6 2024-09-16T13:17:18.121 INFO:teuthology.orchestra.run.smithi098.stdout:1507 trunc from 0x59d42c6 to 0xe8b4cbc 2024-09-16T13:17:18.124 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:17:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:1508 read 0x2652496 thru 0x265a50a (0x8075 bytes) 2024-09-16T13:17:18.139 INFO:teuthology.orchestra.run.smithi098.stdout:1512 writesame 0x5a8325 thru 0x5b3fcc data_size 0x4(0xbca8 bytes) 2024-09-16T13:17:18.153 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:18.148+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:18.158 INFO:teuthology.orchestra.run.smithi098.stdout:1513 write 0x45e7077 thru 0x45f4d0f (0xdc99 bytes) 2024-09-16T13:17:18.163 INFO:teuthology.orchestra.run.smithi098.stdout:1514 read 0x7b993b5 thru 0x7ba2b46 (0x9792 bytes) 2024-09-16T13:17:18.163 INFO:teuthology.orchestra.run.smithi098.stdout:1515 read 0x251cba4 thru 0x252aa90 (0xdeed bytes) 2024-09-16T13:17:18.164 INFO:teuthology.orchestra.run.smithi098.stdout:1516 write 0xa622294 thru 0xa6317a9 (0xf516 bytes) 2024-09-16T13:17:18.170 INFO:teuthology.orchestra.run.smithi098.stdout:1517 write 0xa4ab682 thru 0xa4b03e4 (0x4d63 bytes) 2024-09-16T13:17:18.176 INFO:teuthology.orchestra.run.smithi098.stdout:1519 read 0x1fc8e94 thru 0x1fce53e (0x56ab bytes) 2024-09-16T13:17:18.182 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:18.183 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:18.183 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:18.183 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:18.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:18.197 INFO:teuthology.orchestra.run.smithi098.stdout:1520 trunc from 0xe8b4cbc to 0x39edc10 2024-09-16T13:17:18.225 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.254 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:18.248+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:18.256 INFO:teuthology.orchestra.run.smithi098.stdout:1522 write 0xcd167cc thru 0xcd22110 (0xb945 bytes) 2024-09-16T13:17:18.263 INFO:teuthology.orchestra.run.smithi098.stdout:1523 read 0x6a1a52 thru 0x6a8fae (0x755d bytes) 2024-09-16T13:17:18.265 INFO:teuthology.orchestra.run.smithi098.stdout:1524 read 0x59f2602 thru 0x5a02404 (0xfe03 bytes) 2024-09-16T13:17:18.265 INFO:teuthology.orchestra.run.smithi098.stdout:1525 read 0x898a172 thru 0x8997b82 (0xda11 bytes) 2024-09-16T13:17:18.265 INFO:teuthology.orchestra.run.smithi098.stdout:1526 writesame 0x1abef09 thru 0x1ac4d72 data_size 0x5(0x5e6a bytes) 2024-09-16T13:17:18.275 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.275 INFO:teuthology.orchestra.run.smithi098.stdout:1531 punch from 0x407633 to 0x410542, (0x8f0f bytes) 2024-09-16T13:17:18.276 INFO:teuthology.orchestra.run.smithi098.stdout:1532 write 0x760993a thru 0x7610249 (0x6910 bytes) 2024-09-16T13:17:18.283 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.283 INFO:teuthology.orchestra.run.smithi098.stdout:1534 punch from 0x5e27b4 to 0x5ebda5, (0x95f1 bytes) 2024-09-16T13:17:18.284 INFO:teuthology.orchestra.run.smithi098.stdout:1539 read 0xc6accdc thru 0xc6bc9ca (0xfcef bytes) 2024-09-16T13:17:18.284 INFO:teuthology.orchestra.run.smithi098.stdout:1541 read 0xbdeb8ce thru 0xbdf7f5b (0xc68e bytes) 2024-09-16T13:17:18.284 INFO:teuthology.orchestra.run.smithi098.stdout:1542 punch from 0x252c488 to 0x252c711, (0x289 bytes) 2024-09-16T13:17:18.284 INFO:teuthology.orchestra.run.smithi098.stdout:1543 punch from 0x363f98b to 0x364372e, (0x3da3 bytes) 2024-09-16T13:17:18.284 INFO:teuthology.orchestra.run.smithi098.stdout:1545 write 0x5669bf5 thru 0x5677d9b (0xe1a7 bytes) 2024-09-16T13:17:18.291 INFO:teuthology.orchestra.run.smithi098.stdout:1546 write 0x42ccd2b thru 0x42ce467 (0x173d bytes) 2024-09-16T13:17:18.298 INFO:teuthology.orchestra.run.smithi098.stdout:1547 punch from 0x1517aa6 to 0x151daa3, (0x5ffd bytes) 2024-09-16T13:17:18.298 INFO:teuthology.orchestra.run.smithi098.stdout:1548 read 0x29f98e thru 0x29f9c8 (0x3b bytes) 2024-09-16T13:17:18.299 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:18.299 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:18.299 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:18.299 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:18.299 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:18.300 INFO:teuthology.orchestra.run.smithi098.stdout:1551 writesame 0x61ff8c6 thru 0x620b96e data_size 0x25(0xc0a9 bytes) 2024-09-16T13:17:18.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:18.309 INFO:teuthology.orchestra.run.smithi098.stdout:1554 punch from 0x6e6cf01 to 0x6e74db8, (0x7eb7 bytes) 2024-09-16T13:17:18.309 INFO:teuthology.orchestra.run.smithi098.stdout:1555 write 0xb5b3ef5 thru 0xb5b8ef0 (0x4ffc bytes) 2024-09-16T13:17:18.315 INFO:teuthology.orchestra.run.smithi098.stdout:1556 read 0x132e517 thru 0x13320ee (0x3bd8 bytes) 2024-09-16T13:17:18.316 INFO:teuthology.orchestra.run.smithi098.stdout:1557 writesame 0xa6750c9 thru 0xa68032b data_size 0x1(0xb263 bytes) 2024-09-16T13:17:18.355 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:18.350+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:18.359 INFO:teuthology.orchestra.run.smithi098.stdout:1558 writesame 0x6568b5a thru 0x6573ce4 data_size 0x2b(0xb18b bytes) 2024-09-16T13:17:18.367 INFO:teuthology.orchestra.run.smithi098.stdout:1559 writesame 0xd81fa2 thru 0xd8b4ac data_size 0xd(0x950b bytes) 2024-09-16T13:17:18.373 INFO:teuthology.orchestra.run.smithi098.stdout:1560 punch from 0x51df7c7 to 0x51e80e7, (0x8920 bytes) 2024-09-16T13:17:18.373 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.373 INFO:teuthology.orchestra.run.smithi098.stdout:1562 write 0x4d695f2 thru 0x4d6d31f (0x3d2e bytes) 2024-09-16T13:17:18.380 INFO:teuthology.orchestra.run.smithi098.stdout:1563 writesame 0x26da036 thru 0x26e3e49 data_size 0x43(0x9e14 bytes) 2024-09-16T13:17:18.385 INFO:teuthology.orchestra.run.smithi098.stdout:1564 punch from 0x85a5c89 to 0x85b110a, (0xb481 bytes) 2024-09-16T13:17:18.385 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.385 INFO:teuthology.orchestra.run.smithi098.stdout:1566 read 0x97a1137 thru 0x97a51a1 (0x406b bytes) 2024-09-16T13:17:18.385 INFO:teuthology.orchestra.run.smithi098.stdout:1567 punch from 0x3cde50c to 0x3cea5e4, (0xc0d8 bytes) 2024-09-16T13:17:18.385 INFO:teuthology.orchestra.run.smithi098.stdout:1568 read 0x22c0cd4 thru 0x22c203c (0x1369 bytes) 2024-09-16T13:17:18.386 INFO:teuthology.orchestra.run.smithi098.stdout:1569 read 0x197b4ab thru 0x1984573 (0x90c9 bytes) 2024-09-16T13:17:18.386 INFO:teuthology.orchestra.run.smithi098.stdout:1571 write 0x63a1842 thru 0x63af39a (0xdb59 bytes) 2024-09-16T13:17:18.400 INFO:teuthology.orchestra.run.smithi098.stdout:1572 write 0xd1e0e4f thru 0xd1eb870 (0xaa22 bytes) 2024-09-16T13:17:18.407 INFO:teuthology.orchestra.run.smithi098.stdout:1573 write 0x1a05f63 thru 0x1a14b2f (0xebcd bytes) 2024-09-16T13:17:18.415 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:18.415 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:18.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:18.415 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8262 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:13.928951+0000", 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.479843035, 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017362793000000001, 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.416 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8262 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.928951+0000", 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.928951+0000", 2024-09-16T13:17:18.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.928953+0000", 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2000000000000001e-06 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.928965+0000", 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1834999999999999e-05 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.928989+0000", 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3807000000000001e-05 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929001+0000", 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2313999999999999e-05 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.939520+0000", 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010519404 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.939596+0000", 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5006000000000007e-05 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.939844+0000", 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024883700000000002 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.945949+0000", 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0061049529999999998 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946209+0000", 2024-09-16T13:17:18.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025977299999999998 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946267+0000", 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8183999999999997e-05 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946314+0000", 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6480000000000002e-05 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.422 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8263 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:13.929157+0000", 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.4796373169999999, 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017282163, 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.424 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8263 2024-09-16T13:17:18.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.424 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929157+0000", 2024-09-16T13:17:18.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929157+0000", 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929158+0000", 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7e-06 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929168+0000", 2024-09-16T13:17:18.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5349999999999993e-06 2024-09-16T13:17:18.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929188+0000", 2024-09-16T13:17:18.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9675000000000001e-05 2024-09-16T13:17:18.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929198+0000", 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0426e-05 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.940334+0000", 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011136398 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.940435+0000", 2024-09-16T13:17:18.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100886 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.940788+0000", 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035262800000000001 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.945955+0000", 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051675970000000003 2024-09-16T13:17:18.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946370+0000", 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041468499999999998 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946409+0000", 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9254000000000001e-05 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946439+0000", 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9379000000000001e-05 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8318 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:13.981160+0000", 2024-09-16T13:17:18.430 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.4276343760000003, 2024-09-16T13:17:18.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016937001, 2024-09-16T13:17:18.431 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.431 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.431 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8318 2024-09-16T13:17:18.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.431 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981160+0000", 2024-09-16T13:17:18.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981160+0000", 2024-09-16T13:17:18.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981161+0000", 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.511e-06 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981170+0000", 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4120000000000001e-06 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981194+0000", 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4697000000000001e-05 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981208+0000", 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3723000000000001e-05 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.993490+0000", 2024-09-16T13:17:18.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012281930999999999 2024-09-16T13:17:18.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.993613+0000", 2024-09-16T13:17:18.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012345800000000001 2024-09-16T13:17:18.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.994131+0000", 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051729 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.997478+0000", 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033476479999999999 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.998029+0000", 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00055065099999999998 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.998065+0000", 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6227999999999999e-05 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.998097+0000", 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1452000000000003e-05 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8369 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3524596659999997, 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020058919000000001, 2024-09-16T13:17:18.438 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8369 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:18.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:18.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056335+0000", 2024-09-16T13:17:18.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-16T13:17:18.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056338+0000", 2024-09-16T13:17:18.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:17:18.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056344+0000", 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8150000000000002e-06 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056348+0000", 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.070951+0000", 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014603055 2024-09-16T13:17:18.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071060+0000", 2024-09-16T13:17:18.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109462 2024-09-16T13:17:18.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071508+0000", 2024-09-16T13:17:18.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044824200000000001 2024-09-16T13:17:18.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074304+0000", 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027951719999999998 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076331+0000", 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020276529999999999 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076369+0000", 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7557e-05 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076393+0000", 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4445999999999999e-05 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.445 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8370 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3524423480000003, 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021240851000000002, 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.446 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8370 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:18.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-16T13:17:18.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056355+0000", 2024-09-16T13:17:18.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.667e-06 2024-09-16T13:17:18.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056360+0000", 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6620000000000002e-06 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056366+0000", 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072672+0000", 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016306157000000002 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072746+0000", 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3648999999999999e-05 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.073364+0000", 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061885499999999997 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076445+0000", 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003080396 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077482+0000", 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001036709 2024-09-16T13:17:18.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077535+0000", 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3211000000000003e-05 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077592+0000", 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7664000000000002e-05 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.452 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8371 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3524135529999999, 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021386721000000001, 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.453 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8371 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056381+0000", 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056385+0000", 2024-09-16T13:17:18.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:17:18.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056391+0000", 2024-09-16T13:17:18.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6959999999999999e-06 2024-09-16T13:17:18.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056396+0000", 2024-09-16T13:17:18.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2549999999999999e-06 2024-09-16T13:17:18.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074355+0000", 2024-09-16T13:17:18.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017959098999999999 2024-09-16T13:17:18.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074503+0000", 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014797200000000001 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074955+0000", 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045213300000000002 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077607+0000", 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026519500000000001 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077660+0000", 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077725+0000", 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4991000000000002e-05 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077767+0000", 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.460 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8372 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3523925319999996, 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021557531000000001, 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.461 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056402+0000", 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1399999999999997e-07 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056405+0000", 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6520000000000002e-06 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056411+0000", 2024-09-16T13:17:18.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2009999999999996e-06 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056415+0000", 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.211e-06 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075482+0000", 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019067163000000002 2024-09-16T13:17:18.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075546+0000", 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3516000000000006e-05 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075844+0000", 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029810999999999998 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077784+0000", 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019399350000000001 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077823+0000", 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9524000000000003e-05 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077917+0000", 2024-09-16T13:17:18.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4090000000000002e-05 2024-09-16T13:17:18.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077959+0000", 2024-09-16T13:17:18.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1615000000000001e-05 2024-09-16T13:17:18.467 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.467 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.467 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8373 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3523721890000004, 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022897421000000001, 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.468 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3200000000000005e-07 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056427+0000", 2024-09-16T13:17:18.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-06 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056433+0000", 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056437+0000", 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:17:18.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076470+0000", 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020032239 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076524+0000", 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4475000000000003e-05 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076996+0000", 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000471905 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.078749+0000", 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001752539 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079217+0000", 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046819400000000001 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079276+0000", 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9407000000000002e-05 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079319+0000", 2024-09-16T13:17:18.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8397 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085745+0000", 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3230485219999997, 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017309516, 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8397 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085745+0000", 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085745+0000", 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085746+0000", 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085748+0000", 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.39e-06 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085754+0000", 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3569999999999997e-06 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085757+0000", 2024-09-16T13:17:18.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.174e-06 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.098475+0000", 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012718615000000001 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.098566+0000", 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1135000000000001e-05 2024-09-16T13:17:18.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099043+0000", 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000476942 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102993+0000", 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039494409999999997 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103000+0000", 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5020000000000003e-06 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103030+0000", 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0134e-05 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103055+0000", 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4468999999999999e-05 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8398 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3230083969999997, 2024-09-16T13:17:18.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017439282, 2024-09-16T13:17:18.483 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.483 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.483 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.483 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.483 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.483 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8398 2024-09-16T13:17:18.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.483 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:18.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:18.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085788+0000", 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4719999999999998e-06 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085793+0000", 2024-09-16T13:17:18.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6179999999999997e-06 2024-09-16T13:17:18.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085796+0000", 2024-09-16T13:17:18.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6680000000000001e-06 2024-09-16T13:17:18.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099514+0000", 2024-09-16T13:17:18.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013717895000000001 2024-09-16T13:17:18.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099873+0000", 2024-09-16T13:17:18.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035970799999999998 2024-09-16T13:17:18.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100269+0000", 2024-09-16T13:17:18.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000395382 2024-09-16T13:17:18.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103068+0000", 2024-09-16T13:17:18.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027992680000000002 2024-09-16T13:17:18.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103114+0000", 2024-09-16T13:17:18.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6189000000000001e-05 2024-09-16T13:17:18.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103178+0000", 2024-09-16T13:17:18.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3845999999999998e-05 2024-09-16T13:17:18.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103225+0000", 2024-09-16T13:17:18.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6888999999999998e-05 2024-09-16T13:17:18.490 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.490 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.490 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.490 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8399 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.490 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:18.490 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3229669199999998, 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017551573000000001, 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8399 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:18.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:18.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085828+0000", 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085830+0000", 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8609999999999998e-06 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085835+0000", 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.437e-06 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085837+0000", 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.571e-06 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100950+0000", 2024-09-16T13:17:18.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015112537000000001 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101047+0000", 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6945000000000001e-05 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101468+0000", 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000420794 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103315+0000", 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018474769999999999 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103321+0000", 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103356+0000", 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4764000000000001e-05 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103379+0000", 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3102999999999999e-05 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.497 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8400 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.322954019, 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017960752, 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8400 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.53e-07 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085842+0000", 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2620000000000001e-06 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085847+0000", 2024-09-16T13:17:18.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2119999999999997e-06 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085849+0000", 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.204e-06 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101995+0000", 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016146384999999999 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102100+0000", 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010482199999999999 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102466+0000", 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000365731 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103389+0000", 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092367199999999995 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103702+0000", 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031220400000000001 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103754+0000", 2024-09-16T13:17:18.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2318999999999998e-05 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103801+0000", 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6687999999999998e-05 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.504 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8539 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.1648224640000002, 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018486425000000001, 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8539 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0500000000000003e-07 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243975+0000", 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800000000000002e-06 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243982+0000", 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243986+0000", 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4100000000000001e-06 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258523+0000", 2024-09-16T13:17:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014536604999999999 2024-09-16T13:17:18.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258678+0000", 2024-09-16T13:17:18.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155341 2024-09-16T13:17:18.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.259218+0000", 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053930600000000003 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262325+0000", 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031076799999999998 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262367+0000", 2024-09-16T13:17:18.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1480000000000003e-05 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262417+0000", 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0555e-05 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262458+0000", 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0556999999999998e-05 2024-09-16T13:17:18.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8540 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.164803043, 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018906683000000001, 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8540 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:18.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243992+0000", 2024-09-16T13:17:18.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.75e-07 2024-09-16T13:17:18.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243995+0000", 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1219999999999999e-06 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244001+0000", 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5010000000000001e-06 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244006+0000", 2024-09-16T13:17:18.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999996e-06 2024-09-16T13:17:18.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260137+0000", 2024-09-16T13:17:18.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016131 2024-09-16T13:17:18.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260200+0000", 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3782999999999996e-05 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260598+0000", 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039746600000000002 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262331+0000", 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017327849999999999 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262786+0000", 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045546300000000003 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262851+0000", 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-05 2024-09-16T13:17:18.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262898+0000", 2024-09-16T13:17:18.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7098000000000002e-05 2024-09-16T13:17:18.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.519 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8541 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.1647838200000002, 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019366889000000002, 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8541 2024-09-16T13:17:18.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244011+0000", 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244014+0000", 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.79e-06 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244020+0000", 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6370000000000002e-06 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244025+0000", 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3359999999999997e-06 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261122+0000", 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017097161 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261228+0000", 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105989 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261475+0000", 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024757000000000002 2024-09-16T13:17:18.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263022+0000", 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015466760000000001 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263300+0000", 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027828100000000001 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263344+0000", 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4008999999999998e-05 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263377+0000", 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2832e-05 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.527 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8564 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:18.527 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.266195+0000", 2024-09-16T13:17:18.527 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.1425988990000002, 2024-09-16T13:17:18.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017191598999999998, 2024-09-16T13:17:18.527 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.527 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.528 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8564 2024-09-16T13:17:18.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.528 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266195+0000", 2024-09-16T13:17:18.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266195+0000", 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266195+0000", 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7799999999999997e-07 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266197+0000", 2024-09-16T13:17:18.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1289999999999999e-06 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266201+0000", 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266205+0000", 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2429999999999999e-06 2024-09-16T13:17:18.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.278604+0000", 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012398782000000001 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.278658+0000", 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3844999999999997e-05 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.278910+0000", 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025174300000000002 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.282368+0000", 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034583880000000002 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.283290+0000", 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092164899999999997 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.283354+0000", 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4255000000000001e-05 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.283387+0000", 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2716000000000003e-05 2024-09-16T13:17:18.533 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11776 3.1a 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.89138081300000005, 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027432719000000001, 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.534 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11776 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517415+0000", 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5239999999999999e-06 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526328+0000", 2024-09-16T13:17:18.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089138059999999998 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526362+0000", 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3775000000000001e-05 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526379+0000", 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6835000000000001e-05 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526467+0000", 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7703999999999999e-05 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526653+0000", 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186379 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.527624+0000", 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097080400000000002 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543313+0000", 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015689456000000001 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544696+0000", 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013822979999999999 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544784+0000", 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8419999999999997e-05 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544846+0000", 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1717999999999994e-05 2024-09-16T13:17:18.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:11778 3.1 e40/23 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head v 40'57, mlcod=40'55)", 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.88489678599999999, 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01745787, 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.541 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523898+0000", 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-06 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533067+0000", 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091691120000000001 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533101+0000", 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3733999999999999e-05 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533115+0000", 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3410999999999999e-05 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533194+0000", 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9827000000000007e-05 2024-09-16T13:17:18.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533223+0000", 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8937000000000001e-05 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541297+0000", 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080739850000000005 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541355+0000", 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7708999999999999e-05 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11777 3.1a 3:59cb7a3f:::rbd_object_map.112d21cb8818:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.88239187500000005, 2024-09-16T13:17:18.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018649718999999999, 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11777 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526404+0000", 2024-09-16T13:17:18.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.654e-06 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526409+0000", 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.882e-06 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526424+0000", 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5328999999999998e-05 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526428+0000", 2024-09-16T13:17:18.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6219999999999998e-06 2024-09-16T13:17:18.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542209+0000", 2024-09-16T13:17:18.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015781765999999999 2024-09-16T13:17:18.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542334+0000", 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125148 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542804+0000", 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046944899999999998 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543732+0000", 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092824699999999999 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544929+0000", 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001196522 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545004+0000", 2024-09-16T13:17:18.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4962999999999994e-05 2024-09-16T13:17:18.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545052+0000", 2024-09-16T13:17:18.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8136999999999997e-05 2024-09-16T13:17:18.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.553 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.557 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11779 3.9 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:18.557 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:18.557 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.86187029299999995, 2024-09-16T13:17:18.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020806181999999999, 2024-09-16T13:17:18.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.557 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:18.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.558 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11779 2024-09-16T13:17:18.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.558 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:18.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546926+0000", 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9769999999999999e-06 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552330+0000", 2024-09-16T13:17:18.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054043980000000004 2024-09-16T13:17:18.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552353+0000", 2024-09-16T13:17:18.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2875999999999999e-05 2024-09-16T13:17:18.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552362+0000", 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7299999999999994e-06 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552447+0000", 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5620999999999999e-05 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552710+0000", 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026273300000000002 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.553446+0000", 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073563299999999997 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.559002+0000", 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055564580000000002 2024-09-16T13:17:18.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567615+0000", 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086126680000000004 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567689+0000", 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3839999999999998e-05 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567730+0000", 2024-09-16T13:17:18.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1248000000000002e-05 2024-09-16T13:17:18.564 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.564 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.564 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.564 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.564 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.564 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:18.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:18.567 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:18.558+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:18.568 INFO:teuthology.orchestra.run.smithi098.stdout:1574 trunc from 0xd1eb871 to 0x28ac29b 2024-09-16T13:17:18.568 INFO:teuthology.orchestra.run.smithi098.stdout:1576 trunc from 0x28ac29b to 0x575e7d3 2024-09-16T13:17:18.568 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.568 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.568 INFO:teuthology.orchestra.run.smithi098.stdout:1580 writesame 0xca0ed28 thru 0xca0fc05 data_size 0x2(0xede bytes) 2024-09-16T13:17:18.568 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:18.568 INFO:teuthology.orchestra.run.smithi098.stdout:1583 trunc from 0xca0fc06 to 0xa791475 2024-09-16T13:17:18.569 INFO:teuthology.orchestra.run.smithi098.stdout:1585 write 0x2969822 thru 0x29782e3 (0xeac2 bytes) 2024-09-16T13:17:18.569 INFO:teuthology.orchestra.run.smithi098.stdout:1586 write 0x19bb101 thru 0x19c4931 (0x9831 bytes) 2024-09-16T13:17:18.569 INFO:teuthology.orchestra.run.smithi098.stdout:1587 trunc from 0xa791475 to 0x443b48c 2024-09-16T13:17:18.569 INFO:teuthology.orchestra.run.smithi098.stdout:1588 read 0x287cd98 thru 0x288bfe9 (0xf252 bytes) 2024-09-16T13:17:18.575 INFO:teuthology.orchestra.run.smithi098.stdout:1589 writesame 0x50e88ca thru 0x50f16b5 data_size 0x1f(0x8dec bytes) 2024-09-16T13:17:18.597 INFO:teuthology.orchestra.run.smithi098.stdout:1591 writesame 0x9257f2e thru 0x925daad data_size 0x30(0x5b80 bytes) 2024-09-16T13:17:18.597 INFO:teuthology.orchestra.run.smithi098.stdout:1592 read 0x5721b48 thru 0x572dfc6 (0xc47f bytes) 2024-09-16T13:17:18.598 INFO:teuthology.orchestra.run.smithi098.stdout:1593 read 0x4d94d1a thru 0x4d9c715 (0x79fc bytes) 2024-09-16T13:17:18.609 INFO:teuthology.orchestra.run.smithi098.stdout:1595 writesame 0x93f54d1 thru 0x9400f88 data_size 0x64(0xbab8 bytes) 2024-09-16T13:17:18.610 INFO:teuthology.orchestra.run.smithi098.stdout:1596 read 0x63b0131 thru 0x63b2fab (0x2e7b bytes) 2024-09-16T13:17:18.618 INFO:teuthology.orchestra.run.smithi098.stdout:1597 trunc from 0x9400f89 to 0xce151bd 2024-09-16T13:17:18.633 INFO:teuthology.orchestra.run.smithi098.stdout:1601 clone 9 order 21 su 262144 sc 11 2024-09-16T13:17:18.667 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:18.660+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:18.713 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:18.714 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:18.714 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:18.714 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:18.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:18.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:18.768 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:18.762+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:18.827 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:18.827 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:18.828 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:18.828 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:18.828 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:18.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:18.861 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:18.868 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:18.864+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:18.899 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:18.916 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:18.937 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2520 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.906087461, 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012988767, 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2520 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024618+0000", 2024-09-16T13:17:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0199999999999998e-07 2024-09-16T13:17:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024620+0000", 2024-09-16T13:17:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1150000000000001e-06 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024624+0000", 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.676e-06 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024626+0000", 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6730000000000001e-06 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037443+0000", 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012816869 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037522+0000", 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9060000000000005e-05 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037606+0000", 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4072000000000004e-05 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2521 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2523136~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.906076089000001, 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013552781999999999, 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2521 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024630+0000", 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1629999999999999e-06 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024635+0000", 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5050000000000001e-06 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024637+0000", 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4040000000000002e-06 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037635+0000", 2024-09-16T13:17:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012997754 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037729+0000", 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4043999999999996e-05 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038182+0000", 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045262500000000001 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2522 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3473408~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.906064266, 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013780114, 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2522 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8999999999999998e-07 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024642+0000", 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3230000000000001e-06 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024646+0000", 2024-09-16T13:17:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024648+0000", 2024-09-16T13:17:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4849999999999999e-06 2024-09-16T13:17:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038251+0000", 2024-09-16T13:17:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013602822000000001 2024-09-16T13:17:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038335+0000", 2024-09-16T13:17:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4215999999999995e-05 2024-09-16T13:17:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038421+0000", 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.543e-05 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2523 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.906051321, 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014015147, 2024-09-16T13:17:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2523 2024-09-16T13:17:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:17:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024656+0000", 2024-09-16T13:17:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.131e-06 2024-09-16T13:17:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024660+0000", 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6990000000000001e-06 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024663+0000", 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0290000000000001e-06 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038456+0000", 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013793481 2024-09-16T13:17:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038549+0000", 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2608000000000005e-05 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038669+0000", 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011982800000000001 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2524 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.906037746999999, 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014381392, 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2524 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024668+0000", 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8400000000000003e-07 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024670+0000", 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.849e-06 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024675+0000", 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.451e-06 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024678+0000", 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8739999999999999e-06 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038723+0000", 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014044562999999999 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038811+0000", 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039049+0000", 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023761100000000001 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2525 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 557056~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.906003399999999, 2024-09-16T13:17:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014602618, 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2525 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024705+0000", 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4650000000000001e-06 2024-09-16T13:17:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024711+0000", 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024716+0000", 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1710000000000004e-06 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039096+0000", 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014380184000000001 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039187+0000", 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1065999999999999e-05 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039304+0000", 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117333 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2526 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1507328~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905985231000001, 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014823846, 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2526 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024722+0000", 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024729+0000", 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024733+0000", 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039337+0000", 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014604026000000001 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039427+0000", 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0437000000000002e-05 2024-09-16T13:17:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039544+0000", 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116234 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2527 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1605632~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905967468, 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015041116, 2024-09-16T13:17:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2527 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024738+0000", 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024746+0000", 2024-09-16T13:17:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2130000000000008e-06 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024753+0000", 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024758+0000", 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7670000000000003e-06 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014820533 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039670+0000", 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1825999999999998e-05 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039779+0000", 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108424 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2528 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.90594203, 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015245242000000001, 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2528 2024-09-16T13:17:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024765+0000", 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1040000000000001e-06 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024772+0000", 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-16T13:17:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024776+0000", 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039810+0000", 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015034194000000001 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039897+0000", 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6864e-05 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040008+0000", 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110802 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2529 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2555904~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905922910999999, 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015471574, 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2529 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024785+0000", 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5969999999999999e-06 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024800+0000", 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4681999999999999e-05 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024806+0000", 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:17:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040039+0000", 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015233007999999999 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040143+0000", 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010322799999999999 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040254+0000", 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011092899999999999 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2530 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905892631, 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015667606000000001, 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2530 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024813+0000", 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024815+0000", 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5160000000000001e-06 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024827+0000", 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1915e-05 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024834+0000", 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5180000000000001e-06 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040286+0000", 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015452565999999999 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040375+0000", 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040480+0000", 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105388 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2531 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3538944~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905865415999999, 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015867612999999999, 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2531 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.993 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024843+0000", 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6960000000000001e-06 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024856+0000", 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-05 2024-09-16T13:17:18.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024863+0000", 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6069999999999996e-06 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040513+0000", 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650662999999999 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040599+0000", 2024-09-16T13:17:18.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5439999999999995e-05 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040707+0000", 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108461 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2532 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905834910999999, 2024-09-16T13:17:18.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061912000000001, 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2532 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024871+0000", 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:18.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024874+0000", 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024881+0000", 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0570000000000004e-06 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024886+0000", 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:19.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040742+0000", 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015856808 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040824+0000", 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040932+0000", 2024-09-16T13:17:19.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108378 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2533 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905814385999999, 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016272282999999998, 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:19.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2533 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:19.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024893+0000", 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419999999999999e-06 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:19.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024900+0000", 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024903+0000", 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040965+0000", 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061565999999999 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041052+0000", 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7213000000000007e-05 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041163+0000", 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011066899999999999 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2534 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3604480~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:19.006 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905795855000001, 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016482746999999999, 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2534 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:19.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024910+0000", 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1299999999999999e-07 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024912+0000", 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024920+0000", 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024924+0000", 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.036e-06 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041197+0000", 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016273242 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041282+0000", 2024-09-16T13:17:19.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5013999999999996e-05 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041392+0000", 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109567 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2535 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:19.011 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905775193, 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016691902000000002, 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2535 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:19.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7599999999999997e-07 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:19.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024932+0000", 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024949+0000", 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7053000000000001e-05 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024961+0000", 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041425+0000", 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016463851000000002 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041515+0000", 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9549999999999995e-05 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041622+0000", 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107132 2024-09-16T13:17:19.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2536 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1638400~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905737387, 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016880869999999999, 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:19.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2536 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024971+0000", 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.615e-06 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024988+0000", 2024-09-16T13:17:19.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6912999999999999e-05 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024993+0000", 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041653+0000", 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016660245000000001 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:19.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041743+0000", 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0099999999999995e-05 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041848+0000", 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105404 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2537 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1671168~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:19.020 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905706634, 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017075346000000002, 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2537 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:19.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024999+0000", 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:19.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025002+0000", 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025009+0000", 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025013+0000", 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1840000000000001e-06 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041886+0000", 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016873249999999999 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041971+0000", 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4487000000000006e-05 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042074+0000", 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102847 2024-09-16T13:17:19.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2538 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2621440~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905686463, 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017314057000000001, 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:19.025 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2538 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:19.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:19.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025019+0000", 2024-09-16T13:17:19.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:17:19.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:19.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025021+0000", 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025033+0000", 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1985e-05 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025043+0000", 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3810000000000005e-06 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042107+0000", 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017064471000000001 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042203+0000", 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5810999999999995e-05 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042333+0000", 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012956099999999999 2024-09-16T13:17:19.029 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2539 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.905658734999999, 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017500180000000001, 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:19.030 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2539 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025047+0000", 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025053+0000", 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025058+0000", 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-16T13:17:19.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025061+0000", 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.072e-06 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042365+0000", 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017304885999999998 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042458+0000", 2024-09-16T13:17:19.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2991000000000005e-05 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042546+0000", 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7904000000000004e-05 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:19.034 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:19.036 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:19.036 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:19.036 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:19.036 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:19.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:19.038 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:19.039 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:19.028+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:19.137 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:19.132+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:19.155 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:19.155 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:19.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:19.238 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:19.232+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:19.273 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:19.273 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:19.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:19.339 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:19.334+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:19.386 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:19.386 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:19.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:19.440 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:19.434+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:19.478 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #7, image image_client.0-clone7 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent8 2024-09-16T13:17:19.540 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:19.532+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:19.553 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:19.553 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:19.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:19.596 INFO:teuthology.orchestra.run.smithi098.stdout:1602 trunc from 0xce151bd to 0x5065db 2024-09-16T13:17:19.641 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:19.634+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:19.652 INFO:teuthology.orchestra.run.smithi098.stdout:1606 read 0x2c3fbb thru 0x2c4d99 (0xddf bytes) 2024-09-16T13:17:19.652 INFO:teuthology.orchestra.run.smithi098.stdout:1607 punch from 0x238ae4 to 0x23c705, (0x3c21 bytes) 2024-09-16T13:17:19.664 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:19.664 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:19.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:19.675 INFO:teuthology.orchestra.run.smithi098.stdout:1608 trunc from 0x5065db to 0xc92da7e 2024-09-16T13:17:19.686 INFO:teuthology.orchestra.run.smithi098.stdout:1609 write 0x2112919 thru 0x211ea68 (0xc150 bytes) 2024-09-16T13:17:19.693 INFO:teuthology.orchestra.run.smithi098.stdout:1610 writesame 0x127dcd4 thru 0x1289ab2 data_size 0x1(0xbddf bytes) 2024-09-16T13:17:19.741 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:19.741 INFO:teuthology.orchestra.run.smithi098.stdout:1613 punch from 0x5de17f1 to 0x5de5a83, (0x4292 bytes) 2024-09-16T13:17:19.741 INFO:teuthology.orchestra.run.smithi098.stdout:1614 read 0x47e7d1e thru 0x47ea93b (0x2c1e bytes) 2024-09-16T13:17:19.742 INFO:teuthology.orchestra.run.smithi098.stdout:1615 writesame 0xc1ff59c thru 0xc2064d4 data_size 0x1(0x6f39 bytes) 2024-09-16T13:17:19.742 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:19.738+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:19.781 INFO:teuthology.orchestra.run.smithi098.stdout:1616 trunc from 0xc92da7e to 0x6d53187 2024-09-16T13:17:19.810 INFO:teuthology.orchestra.run.smithi098.stdout:1619 read 0x7be770 thru 0x7c686d (0x80fe bytes) 2024-09-16T13:17:19.812 INFO:teuthology.orchestra.run.smithi098.stdout:1623 write 0x218999e thru 0x2197fd0 (0xe633 bytes) 2024-09-16T13:17:19.819 INFO:teuthology.orchestra.run.smithi098.stdout:1624 write 0x3a30442 thru 0x3a34fdd (0x4b9c bytes) 2024-09-16T13:17:19.819 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:19.819 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:19.825 INFO:teuthology.orchestra.run.smithi098.stdout:1625 read 0x623bc32 thru 0x6246667 (0xaa36 bytes) 2024-09-16T13:17:19.825 INFO:teuthology.orchestra.run.smithi098.stdout:1626 read 0x67dc10e thru 0x67dcbea (0xadd bytes) 2024-09-16T13:17:19.825 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:19.825 INFO:teuthology.orchestra.run.smithi098.stdout:1628 punch from 0x309099e to 0x309de14, (0xd476 bytes) 2024-09-16T13:17:19.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:19.854 INFO:teuthology.orchestra.run.smithi098.stdout:1629 write 0xe551fa4 thru 0xe55604f (0x40ac bytes) 2024-09-16T13:17:19.854 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:19.848+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:19.861 INFO:teuthology.orchestra.run.smithi098.stdout:1630 writesame 0xbe94436 thru 0xbe9df27 data_size 0x21(0x9af2 bytes) 2024-09-16T13:17:19.867 INFO:teuthology.orchestra.run.smithi098.stdout:1631 write 0x930f43a thru 0x9317ec1 (0x8a88 bytes) 2024-09-16T13:17:19.886 INFO:teuthology.orchestra.run.smithi098.stdout:1632 writesame 0xe7d9653 thru 0xe7e29bf data_size 0x2f(0x936d bytes) 2024-09-16T13:17:19.908 INFO:teuthology.orchestra.run.smithi098.stdout:1633 trunc from 0xe7e29c0 to 0x245b986 2024-09-16T13:17:19.929 INFO:teuthology.orchestra.run.smithi098.stdout:1634 read 0xfbc05a thru 0xfbedb5 (0x2d5c bytes) 2024-09-16T13:17:19.930 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:19.930 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:19.930 INFO:teuthology.orchestra.run.smithi098.stdout:1637 read 0xc5a99 thru 0xc7ca5 (0x220d bytes) 2024-09-16T13:17:19.931 INFO:teuthology.orchestra.run.smithi098.stdout:1638 write 0xf364a5 thru 0xf437a0 (0xd2fc bytes) 2024-09-16T13:17:19.935 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:19.935 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:19.943 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:19.943 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:19.943 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:19.938+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:19.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:19.964 INFO:teuthology.orchestra.run.smithi098.stdout:1642 write 0xc57e2a3 thru 0xc57ea09 (0x767 bytes) 2024-09-16T13:17:19.982 INFO:teuthology.orchestra.run.smithi098.stdout:1643 write 0xcc3f76d thru 0xcc4eef7 (0xf78b bytes) 2024-09-16T13:17:19.990 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:19.990 INFO:teuthology.orchestra.run.smithi098.stdout:1645 write 0x922f6f2 thru 0x92308d3 (0x11e2 bytes) 2024-09-16T13:17:19.996 INFO:teuthology.orchestra.run.smithi098.stdout:1646 writesame 0x6ab2228 thru 0x6ab2ea2 data_size 0x2d(0xc7b bytes) 2024-09-16T13:17:20.001 INFO:teuthology.orchestra.run.smithi098.stdout:1647 writesame 0xaa44dad thru 0xaa4fb36 data_size 0x61(0xad8a bytes) 2024-09-16T13:17:20.007 INFO:teuthology.orchestra.run.smithi098.stdout:1649 read 0x7eecb39 thru 0x7eedb5d (0x1025 bytes) 2024-09-16T13:17:20.021 INFO:teuthology.orchestra.run.smithi098.stdout:1650 write 0xd8e10d1 thru 0xd8ea4ba (0x93ea bytes) 2024-09-16T13:17:20.028 INFO:teuthology.orchestra.run.smithi098.stdout:1651 read 0x3c9a2c6 thru 0x3ca05df (0x631a bytes) 2024-09-16T13:17:20.044 INFO:teuthology.orchestra.run.smithi098.stdout:1653 trunc from 0xd8ea4bb to 0x5ca97b6 2024-09-16T13:17:20.044 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:20.040+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:20.069 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.074 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:20.074 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:20.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:20.091 INFO:teuthology.orchestra.run.smithi098.stdout:1656 write 0xb745be9 thru 0xb755b2d (0xff45 bytes) 2024-09-16T13:17:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.099 INFO:teuthology.orchestra.run.smithi098.stdout:1659 write 0x85d774c thru 0x85e07a5 (0x905a bytes) 2024-09-16T13:17:20.110 INFO:teuthology.orchestra.run.smithi098.stdout:1660 trunc from 0xb755b2e to 0x8edb451 2024-09-16T13:17:20.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:17:20.151 INFO:teuthology.orchestra.run.smithi098.stdout:1661 writesame 0xadc25b1 thru 0xadcf15a data_size 0x83(0xcbaa bytes) 2024-09-16T13:17:20.151 INFO:teuthology.orchestra.run.smithi098.stdout:1662 read 0x35b9896 thru 0x35c6bd8 (0xd343 bytes) 2024-09-16T13:17:20.151 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:20.144+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:20.164 INFO:teuthology.orchestra.run.smithi098.stdout:1665 write 0xc1611e0 thru 0xc16a2c4 (0x90e5 bytes) 2024-09-16T13:17:20.169 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.169 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.169 INFO:teuthology.orchestra.run.smithi098.stdout:1669 punch from 0xb3673be to 0xb36a2e8, (0x2f2a bytes) 2024-09-16T13:17:20.170 INFO:teuthology.orchestra.run.smithi098.stdout:1671 writesame 0x18396fd thru 0x183f061 data_size 0x17(0x5965 bytes) 2024-09-16T13:17:20.177 INFO:teuthology.orchestra.run.smithi098.stdout:1673 write 0x6d07738 thru 0x6d13d61 (0xc62a bytes) 2024-09-16T13:17:20.185 INFO:teuthology.orchestra.run.smithi098.stdout:1674 punch from 0x329b420 to 0x329dae1, (0x26c1 bytes) 2024-09-16T13:17:20.185 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.185 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.195 INFO:teuthology.orchestra.run.smithi098.stdout:1679 write 0xc6ba67e thru 0xc6c1c6e (0x75f1 bytes) 2024-09-16T13:17:20.204 INFO:teuthology.orchestra.run.smithi098.stdout:1680 read 0xb2fd5f4 thru 0xb30a670 (0xd07d bytes) 2024-09-16T13:17:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:20.211 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:20.219 INFO:teuthology.orchestra.run.smithi098.stdout:1681 trunc from 0xc6c1c6f to 0x3b5853b 2024-09-16T13:17:20.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:20.241 INFO:teuthology.orchestra.run.smithi098.stdout:1682 trunc from 0x3b5853b to 0x34db493 2024-09-16T13:17:20.253 INFO:teuthology.orchestra.run.smithi098.stdout:1683 read 0x472342 thru 0x478ff7 (0x6cb6 bytes) 2024-09-16T13:17:20.253 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:20.248+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:20.283 INFO:teuthology.orchestra.run.smithi098.stdout:1685 write 0xd242c27 thru 0xd24c9db (0x9db5 bytes) 2024-09-16T13:17:20.290 INFO:teuthology.orchestra.run.smithi098.stdout:1687 punch from 0x5770731 to 0x57799ce, (0x929d bytes) 2024-09-16T13:17:20.290 INFO:teuthology.orchestra.run.smithi098.stdout:1689 writesame 0x2de8100 thru 0x2de8279 data_size 0xe(0x17a bytes) 2024-09-16T13:17:20.297 INFO:teuthology.orchestra.run.smithi098.stdout:1690 writesame 0xc76db2c thru 0xc77be94 data_size 0x1(0xe369 bytes) 2024-09-16T13:17:20.345 INFO:teuthology.orchestra.run.smithi098.stdout:1691 write 0x120b006 thru 0x1210ba1 (0x5b9c bytes) 2024-09-16T13:17:20.346 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:20.342+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:20.349 INFO:teuthology.orchestra.run.smithi098.stdout:1692 write 0xa6bbfc9 thru 0xa6bde79 (0x1eb1 bytes) 2024-09-16T13:17:20.356 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:20.356 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:20.356 INFO:teuthology.orchestra.run.smithi098.stdout:1693 punch from 0x48d3fb4 to 0x48da683, (0x66cf bytes) 2024-09-16T13:17:20.356 INFO:teuthology.orchestra.run.smithi098.stdout:1694 read 0xabbea89 thru 0xabc8986 (0x9efe bytes) 2024-09-16T13:17:20.356 INFO:teuthology.orchestra.run.smithi098.stdout:1696 write 0xa54ac8e thru 0xa54eacd (0x3e40 bytes) 2024-09-16T13:17:20.362 INFO:teuthology.orchestra.run.smithi098.stdout:1697 write 0x41d6121 thru 0x41d97f0 (0x36d0 bytes) 2024-09-16T13:17:20.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:20.368 INFO:teuthology.orchestra.run.smithi098.stdout:1699 punch from 0xcb7aaf to 0xcc282f, (0xad80 bytes) 2024-09-16T13:17:20.368 INFO:teuthology.orchestra.run.smithi098.stdout:1700 punch from 0x766d4d7 to 0x7674ac6, (0x75ef bytes) 2024-09-16T13:17:20.368 INFO:teuthology.orchestra.run.smithi098.stdout:1701 write 0xc5fd6f7 thru 0xc600614 (0x2f1e bytes) 2024-09-16T13:17:20.376 INFO:teuthology.orchestra.run.smithi098.stdout:1702 read 0x32fe6b2 thru 0x3304296 (0x5be5 bytes) 2024-09-16T13:17:20.394 INFO:teuthology.orchestra.run.smithi098.stdout:1704 trunc from 0xd24c9dc to 0x25a1da7 2024-09-16T13:17:20.436 INFO:teuthology.orchestra.run.smithi098.stdout:1705 write 0x75fe0da thru 0x7603c8c (0x5bb3 bytes) 2024-09-16T13:17:20.447 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:20.440+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:20.459 INFO:teuthology.orchestra.run.smithi098.stdout:1706 write 0x96e0ba7 thru 0x96eff28 (0xf382 bytes) 2024-09-16T13:17:20.466 INFO:teuthology.orchestra.run.smithi098.stdout:1707 read 0x39595ff thru 0x395a18a (0xb8c bytes) 2024-09-16T13:17:20.466 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.466 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.466 INFO:teuthology.orchestra.run.smithi098.stdout:1712 punch from 0x14a6ae2 to 0x14b6874, (0xfd92 bytes) 2024-09-16T13:17:20.466 INFO:teuthology.orchestra.run.smithi098.stdout:1713 read 0x39c0163 thru 0x39cfc34 (0xfad2 bytes) 2024-09-16T13:17:20.466 INFO:teuthology.orchestra.run.smithi098.stdout:1714 read 0x298e2ec thru 0x298e5e4 (0x2f9 bytes) 2024-09-16T13:17:20.466 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.466 INFO:teuthology.orchestra.run.smithi098.stdout:1718 punch from 0x2ba3fe3 to 0x2ba684e, (0x286b bytes) 2024-09-16T13:17:20.466 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.466 INFO:teuthology.orchestra.run.smithi098.stdout:1720 read 0x7eeaa0 thru 0x7ef5ed (0xb4e bytes) 2024-09-16T13:17:20.467 INFO:teuthology.orchestra.run.smithi098.stdout:1721 punch from 0xda18b1 to 0xda2a7b, (0x11ca bytes) 2024-09-16T13:17:20.467 INFO:teuthology.orchestra.run.smithi098.stdout:1722 read 0x67bebbe thru 0x67c90ec (0xa52f bytes) 2024-09-16T13:17:20.467 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:20.467 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:20.468 INFO:teuthology.orchestra.run.smithi098.stdout:1723 write 0x26b2b48 thru 0x26b8450 (0x5909 bytes) 2024-09-16T13:17:20.469 INFO:teuthology.orchestra.run.smithi098.stdout:1724 read 0xe427a thru 0xefdb2 (0xbb39 bytes) 2024-09-16T13:17:20.470 INFO:teuthology.orchestra.run.smithi098.stdout:1725 punch from 0x39ba473 to 0x39c07d7, (0x6364 bytes) 2024-09-16T13:17:20.471 INFO:teuthology.orchestra.run.smithi098.stdout:1726 writesame 0x108d3b3 thru 0x1098c90 data_size 0x1(0xb8de bytes) 2024-09-16T13:17:20.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:20.513 INFO:teuthology.orchestra.run.smithi098.stdout:1727 read 0x7eebe35 thru 0x7ef9889 (0xda55 bytes) 2024-09-16T13:17:20.513 INFO:teuthology.orchestra.run.smithi098.stdout:1728 write 0x89c09b8 thru 0x89c5a72 (0x50bb bytes) 2024-09-16T13:17:20.521 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:20.521 INFO:teuthology.orchestra.run.smithi098.stdout:1730 read 0x49a901a thru 0x49abafc (0x2ae3 bytes) 2024-09-16T13:17:20.521 INFO:teuthology.orchestra.run.smithi098.stdout:1732 punch from 0x60cb891 to 0x60d51dd, (0x994c bytes) 2024-09-16T13:17:20.521 INFO:teuthology.orchestra.run.smithi098.stdout:1735 punch from 0x51a8b6 to 0x51d356, (0x2aa0 bytes) 2024-09-16T13:17:20.521 INFO:teuthology.orchestra.run.smithi098.stdout:1736 punch from 0x7b9bdeb to 0x7b9d591, (0x17a6 bytes) 2024-09-16T13:17:20.522 INFO:teuthology.orchestra.run.smithi098.stdout:1738 read 0x56112f thru 0x56d25c (0xc12e bytes) 2024-09-16T13:17:20.522 INFO:teuthology.orchestra.run.smithi098.stdout:1739 read 0xfb2ba1 thru 0xfb47c0 (0x1c20 bytes) 2024-09-16T13:17:20.523 INFO:teuthology.orchestra.run.smithi098.stdout:1740 punch from 0x76e3eea to 0x76ee8ec, (0xaa02 bytes) 2024-09-16T13:17:20.524 INFO:teuthology.orchestra.run.smithi098.stdout:1741 write 0x9121d63 thru 0x9124385 (0x2623 bytes) 2024-09-16T13:17:20.530 INFO:teuthology.orchestra.run.smithi098.stdout:1742 read 0x7f5b0f thru 0x7ffafc (0x9fee bytes) 2024-09-16T13:17:20.548 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:20.540+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:20.551 INFO:teuthology.orchestra.run.smithi098.stdout:1743 write 0xb7dd249 thru 0xb7eadab (0xdb63 bytes) 2024-09-16T13:17:20.555 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:20.555 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:20.555 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:20.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-09-16T13:17:20.557 INFO:teuthology.orchestra.run.smithi098.stdout:1744 read 0x69f200 thru 0x6a8977 (0x9778 bytes) 2024-09-16T13:17:20.558 INFO:teuthology.orchestra.run.smithi098.stdout:1748 read 0x686b310 thru 0x6879c31 (0xe922 bytes) 2024-09-16T13:17:20.558 INFO:teuthology.orchestra.run.smithi098.stdout:1749 trunc from 0xb7eadac to 0xb84b516 2024-09-16T13:17:20.570 INFO:teuthology.orchestra.run.smithi098.stdout:1752 writesame 0x8399050 thru 0x8399790 data_size 0x1(0x741 bytes) 2024-09-16T13:17:20.576 INFO:teuthology.orchestra.run.smithi098.stdout:1753 punch from 0xa04e45e to 0xa05c824, (0xe3c6 bytes) 2024-09-16T13:17:20.588 INFO:teuthology.orchestra.run.smithi098.stdout:1754 trunc from 0xb84b516 to 0x520713d 2024-09-16T13:17:20.615 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:20.616 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:20.623 INFO:teuthology.orchestra.run.smithi098.stdout:1755 write 0x5a93189 thru 0x5a95a3b (0x28b3 bytes) 2024-09-16T13:17:20.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:20.653 INFO:teuthology.orchestra.run.smithi098.stdout:1757 writesame 0xda425ae thru 0xda4e839 data_size 0x2(0xc28c bytes) 2024-09-16T13:17:20.653 INFO:teuthology.orchestra.run.smithi098.stdout:1759 read 0x4f623f6 thru 0x4f6e137 (0xbd42 bytes) 2024-09-16T13:17:20.653 INFO:teuthology.orchestra.run.smithi098.stdout:1760 write 0x68c92c7 thru 0x68d2c90 (0x99ca bytes) 2024-09-16T13:17:20.654 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:20.648+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:20.657 INFO:teuthology.orchestra.run.smithi098.stdout:1762 read 0x1c632d1 thru 0x1c66b5c (0x388c bytes) 2024-09-16T13:17:20.657 INFO:teuthology.orchestra.run.smithi098.stdout:1763 write 0x1bca265 thru 0x1bd9efe (0xfc9a bytes) 2024-09-16T13:17:20.662 INFO:teuthology.orchestra.run.smithi098.stdout:1764 write 0x35b9e4e thru 0x35c84b4 (0xe667 bytes) 2024-09-16T13:17:20.672 INFO:teuthology.orchestra.run.smithi098.stdout:1765 write 0x99d6237 thru 0x99e2b7c (0xc946 bytes) 2024-09-16T13:17:20.688 INFO:teuthology.orchestra.run.smithi098.stdout:1766 trunc from 0xda4e83a to 0x77eab47 2024-09-16T13:17:20.706 INFO:teuthology.orchestra.run.smithi098.stdout:1769 read 0x4918b54 thru 0x49272e2 (0xe78f bytes) 2024-09-16T13:17:20.707 INFO:teuthology.orchestra.run.smithi098.stdout:1770 read 0x4226dd3 thru 0x422b496 (0x46c4 bytes) 2024-09-16T13:17:20.707 INFO:teuthology.orchestra.run.smithi098.stdout:1771 write 0x5a2dd78 thru 0x5a38ae2 (0xad6b bytes) 2024-09-16T13:17:20.731 INFO:teuthology.orchestra.run.smithi098.stdout:1772 writesame 0xaa1b8cd thru 0xaa282b7 data_size 0x1(0xc9eb bytes) 2024-09-16T13:17:20.731 INFO:teuthology.orchestra.run.smithi098.stdout:1773 punch from 0x3ef1b5c to 0x3efbdfa, (0xa29e bytes) 2024-09-16T13:17:20.731 INFO:teuthology.orchestra.run.smithi098.stdout:1774 read 0x2a8b77f thru 0x2a97eda (0xc75c bytes) 2024-09-16T13:17:20.731 INFO:teuthology.orchestra.run.smithi098.stdout:1775 writesame 0x6f6f0ed thru 0x6f70725 data_size 0x1(0x1639 bytes) 2024-09-16T13:17:20.742 INFO:teuthology.orchestra.run.smithi098.stdout:1777 punch from 0x4851952 to 0x48610c5, (0xf773 bytes) 2024-09-16T13:17:20.742 INFO:teuthology.orchestra.run.smithi098.stdout:1778 write 0x316f206 thru 0x317dd8b (0xeb86 bytes) 2024-09-16T13:17:20.749 INFO:teuthology.orchestra.run.smithi098.stdout:1781 clone 10 order 24 su 1048576 sc 8 2024-09-16T13:17:20.749 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:20.744+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:20.752 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:20.752 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:20.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:20.850 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:20.846+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:20.858 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-09-16T13:17:20.852+0000 7f1b776ba700 -1 mon.a@0(leader).osd e42 definitely_dead 0 2024-09-16T13:17:20.900 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:20.900 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:20.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:20.951 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:20.946+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:21.010 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:21.010 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:21.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:21.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:21.068 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:21.064+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:21.124 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:21.125 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:21.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:21.153 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:21.142+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:21.164 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:17:21.166 INFO:teuthology.orchestra.run.smithi098.stderr:marked out osd.0. 2024-09-16T13:17:21.211 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:21.229 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:21.247 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:21.247 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:21.248 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:21.255 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:21.248+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:21.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:21.268 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:21.293 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:21.315 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:21.338 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:21.356 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:21.350+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:21.361 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:21.363 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:21.363 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:21.363 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:21.364 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:21.364 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:21.364 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:21.364 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:21.364 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:21.368 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5101 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.582695102000001, 2024-09-16T13:17:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595248, 2024-09-16T13:17:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5101 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777333+0000", 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1949999999999996e-06 2024-09-16T13:17:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777342+0000", 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6950000000000006e-06 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777348+0000", 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785241+0000", 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078929329999999995 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785391+0000", 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150113 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785969+0000", 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057822899999999998 2024-09-16T13:17:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789745+0000", 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037756529999999999 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793823+0000", 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040786299999999998 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793880+0000", 2024-09-16T13:17:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7157999999999997e-05 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793924+0000", 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3407e-05 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5102 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.582647809999999, 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016663710000000002, 2024-09-16T13:17:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5102 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777377+0000", 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4600000000000003e-07 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777381+0000", 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5290000000000002e-06 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777393+0000", 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2001000000000001e-05 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777401+0000", 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3869999999999997e-06 2024-09-16T13:17:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786753+0000", 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093523649999999996 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786860+0000", 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106921 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787286+0000", 2024-09-16T13:17:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042651800000000003 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793815+0000", 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065290579999999999 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793932+0000", 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116688 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794002+0000", 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0313000000000003e-05 2024-09-16T13:17:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794040+0000", 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7083999999999999e-05 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5103 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.582586923999999, 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017505544000000001, 2024-09-16T13:17:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5103 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777438+0000", 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.78e-07 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777442+0000", 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5950000000000001e-06 2024-09-16T13:17:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777453+0000", 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0372e-05 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777458+0000", 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6559999999999997e-06 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788034+0000", 2024-09-16T13:17:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010575753 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788111+0000", 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7038000000000003e-05 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788463+0000", 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000352028 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794263+0000", 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005800342 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794837+0000", 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057334300000000003 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794897+0000", 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0690000000000003e-05 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794942+0000", 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4848999999999997e-05 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5104 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.582542516, 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017694142999999999, 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5104 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777482+0000", 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777487+0000", 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6639999999999997e-06 2024-09-16T13:17:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777496+0000", 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2890000000000003e-06 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777501+0000", 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2730000000000002e-06 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788951+0000", 2024-09-16T13:17:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011449429000000001 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789007+0000", 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5643999999999998e-05 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789205+0000", 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019820800000000001 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794268+0000", 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050636850000000001 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795094+0000", 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082569700000000002 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795139+0000", 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5055000000000001e-05 2024-09-16T13:17:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795175+0000", 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6149000000000002e-05 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5105 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.582480175000001, 2024-09-16T13:17:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017729482000000001, 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5105 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777544+0000", 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1100000000000004e-07 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777549+0000", 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777561+0000", 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1524e-05 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777566+0000", 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.558e-06 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789781+0000", 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012214981 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789870+0000", 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9171000000000004e-05 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790313+0000", 2024-09-16T13:17:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044255299999999999 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794273+0000", 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039596129999999998 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795213+0000", 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094086099999999998 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795239+0000", 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5324e-05 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795273+0000", 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4198e-05 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5106 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.582435030999999, 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018621867, 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5106 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777590+0000", 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777594+0000", 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6750000000000001e-06 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777604+0000", 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0076e-05 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777610+0000", 2024-09-16T13:17:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7529999999999998e-06 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790893+0000", 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013282845 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790999+0000", 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106289 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791493+0000", 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049357800000000005 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794662+0000", 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031691409999999999 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796146+0000", 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00148375 2024-09-16T13:17:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796176+0000", 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0349e-05 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796211+0000", 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4530000000000003e-05 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5107 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.582403769999999, 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018745067000000001, 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5107 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777621+0000", 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-07 2024-09-16T13:17:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777624+0000", 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7950000000000001e-06 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777633+0000", 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5979999999999997e-06 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777638+0000", 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.989e-06 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792024+0000", 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014385692 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792104+0000", 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0567000000000003e-05 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792340+0000", 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023624200000000001 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794667+0000", 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023269990000000002 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796304+0000", 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001636981 2024-09-16T13:17:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796331+0000", 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6492999999999999e-05 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796365+0000", 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4102e-05 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5108 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.582349941, 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019090617000000001, 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5108 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777677+0000", 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7879999999999998e-06 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777686+0000", 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7679999999999997e-06 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777689+0000", 2024-09-16T13:17:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.123e-06 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792751+0000", 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01506158 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792855+0000", 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010456 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793305+0000", 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045006100000000003 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794671+0000", 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001366304 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796393+0000", 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001721272 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796733+0000", 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000340611 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796764+0000", 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.099e-05 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5208 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.480590490000001, 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014882177999999999, 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5208 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879434+0000", 2024-09-16T13:17:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.322e-06 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879443+0000", 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879459+0000", 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6133999999999998e-05 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879467+0000", 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2400000000000007e-06 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888339+0000", 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088721619999999994 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888461+0000", 2024-09-16T13:17:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121341 2024-09-16T13:17:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888819+0000", 2024-09-16T13:17:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035782799999999998 2024-09-16T13:17:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891271+0000", 2024-09-16T13:17:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002452872 2024-09-16T13:17:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894224+0000", 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029522910000000001 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894275+0000", 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1647999999999997e-05 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894315+0000", 2024-09-16T13:17:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9993999999999997e-05 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5210 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.480035301999999, 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01466423, 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5210 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879989+0000", 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8800000000000001e-07 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879995+0000", 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880017+0000", 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2101000000000001e-05 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880024+0000", 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9240000000000001e-06 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890250+0000", 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010226012 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890344+0000", 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.446e-05 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890770+0000", 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042575099999999998 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894344+0000", 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003574163 2024-09-16T13:17:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894549+0000", 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020477599999999999 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894606+0000", 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6891000000000001e-05 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894653+0000", 2024-09-16T13:17:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6867999999999999e-05 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5211 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.479991239, 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014800604, 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5211 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880033+0000", 2024-09-16T13:17:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.06e-07 2024-09-16T13:17:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880039+0000", 2024-09-16T13:17:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880052+0000", 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-05 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880059+0000", 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3090000000000001e-06 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891316+0000", 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011257118999999999 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891374+0000", 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8020999999999999e-05 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891577+0000", 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020371699999999999 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894348+0000", 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027708300000000002 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894726+0000", 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037819399999999999 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894784+0000", 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7667999999999997e-05 2024-09-16T13:17:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894833+0000", 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8936e-05 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5212 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.479635865000001, 2024-09-16T13:17:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014645704000000001, 2024-09-16T13:17:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5212 2024-09-16T13:17:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880389+0000", 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1680000000000001e-06 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880395+0000", 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880408+0000", 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.308e-05 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880414+0000", 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.105e-06 2024-09-16T13:17:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892037+0000", 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011622402 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892116+0000", 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9939999999999997e-05 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892597+0000", 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048076500000000002 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894492+0000", 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001895246 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894921+0000", 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000428114 2024-09-16T13:17:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894985+0000", 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4777999999999995e-05 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895034+0000", 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8133999999999997e-05 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5213 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.479502715000001, 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014632608, 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5213 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880522+0000", 2024-09-16T13:17:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2600000000000001e-07 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880527+0000", 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880538+0000", 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917999999999999e-05 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880543+0000", 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.361e-06 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893069+0000", 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012525497999999999 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893152+0000", 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2805000000000004e-05 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893635+0000", 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048315100000000002 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894852+0000", 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001216806 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895075+0000", 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022300299999999999 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895122+0000", 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7077999999999999e-05 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895154+0000", 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1919999999999999e-05 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5214 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.479348064, 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017785977000000001, 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5214 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880681+0000", 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880689+0000", 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3610000000000001e-06 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880694+0000", 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5809999999999998e-06 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895186+0000", 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014492430000000001 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895311+0000", 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012518300000000001 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895760+0000", 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044864900000000002 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898337+0000", 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025765229999999998 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898354+0000", 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7538999999999999e-05 2024-09-16T13:17:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898406+0000", 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2054e-05 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898462+0000", 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5404e-05 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5215 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.479083230000001, 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018191319000000001, 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5215 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880941+0000", 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880946+0000", 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3710000000000002e-06 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880955+0000", 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1230000000000006e-06 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880959+0000", 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896782+0000", 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015822716000000001 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896876+0000", 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3667999999999998e-05 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.897289+0000", 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041357399999999999 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898474+0000", 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011847209999999999 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899036+0000", 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056215699999999998 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899090+0000", 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3662999999999998e-05 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899132+0000", 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1814000000000003e-05 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5282 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.410411116000001, 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014924988, 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5282 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.53e-07 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949635+0000", 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2422999999999999e-05 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949641+0000", 2024-09-16T13:17:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.896e-06 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949644+0000", 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5270000000000001e-06 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959656+0000", 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010012378000000001 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959747+0000", 2024-09-16T13:17:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1273000000000005e-05 2024-09-16T13:17:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960045+0000", 2024-09-16T13:17:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029733 2024-09-16T13:17:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963196+0000", 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031512620000000002 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964453+0000", 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001257357 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964502+0000", 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9044e-05 2024-09-16T13:17:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964538+0000", 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5244999999999999e-05 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5283 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.410375962, 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015065762, 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5283 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949650+0000", 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.736e-06 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949653+0000", 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2969999999999999e-06 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949655+0000", 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1890000000000001e-06 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960643+0000", 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010988015 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960732+0000", 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9097000000000004e-05 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961195+0000", 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000462292 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963208+0000", 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002013641 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964585+0000", 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013765769999999999 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964669+0000", 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3863000000000007e-05 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964713+0000", 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4706999999999997e-05 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5284 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.410366093, 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015152802, 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5284 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5899999999999998e-07 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949659+0000", 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.601e-06 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949672+0000", 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2364999999999999e-05 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949674+0000", 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961869+0000", 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012194903999999999 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961971+0000", 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102154 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.962502+0000", 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00053037500000000003 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964751+0000", 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022487800000000001 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964755+0000", 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964789+0000", 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4054000000000003e-05 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964810+0000", 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0922000000000001e-05 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5285 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.410347175, 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015553022, 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5285 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949677+0000", 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6899999999999999e-07 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949679+0000", 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0090000000000002e-06 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949682+0000", 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2550000000000002e-06 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949685+0000", 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.621e-06 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963409+0000", 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013724485999999999 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963497+0000", 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7560999999999997e-05 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963933+0000", 2024-09-16T13:17:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043592700000000002 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964820+0000", 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088772299999999998 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965165+0000", 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000344426 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965201+0000", 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5967000000000003e-05 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965230+0000", 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11223 3.8 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 33927~20122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.8991748450000001, 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019333487, 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11223 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460851+0000", 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5730000000000002e-06 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464824+0000", 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039725730000000001 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464861+0000", 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7233000000000001e-05 2024-09-16T13:17:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464872+0000", 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209000000000001e-05 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464987+0000", 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011423 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465188+0000", 2024-09-16T13:17:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020091600000000001 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465929+0000", 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00074102 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.475697+0000", 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009768367 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480068+0000", 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043713290000000002 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480143+0000", 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5000999999999996e-05 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480182+0000", 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9035999999999997e-05 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:21.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:21.494 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:21.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:21.497 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:21.490+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:21.596 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:21.592+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:21.602 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:21.603 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:21.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:21.697 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:21.692+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:21.708 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:21.708 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:21.708 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:21.708 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:21.709 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:21.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:21.798 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:21.792+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8262 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:13.928951+0000", 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8868218839999997, 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017362793000000001, 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8262 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.928951+0000", 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.928951+0000", 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.928953+0000", 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2000000000000001e-06 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.928965+0000", 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1834999999999999e-05 2024-09-16T13:17:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.928989+0000", 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3807000000000001e-05 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929001+0000", 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2313999999999999e-05 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.939520+0000", 2024-09-16T13:17:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010519404 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.939596+0000", 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5006000000000007e-05 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.939844+0000", 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024883700000000002 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.945949+0000", 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0061049529999999998 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946209+0000", 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025977299999999998 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946267+0000", 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8183999999999997e-05 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946314+0000", 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6480000000000002e-05 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8263 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:13.929157+0000", 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.8866161659999996, 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017282163, 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8263 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929157+0000", 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929157+0000", 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929158+0000", 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7e-06 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929168+0000", 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5349999999999993e-06 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929188+0000", 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9675000000000001e-05 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.929198+0000", 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0426e-05 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.940334+0000", 2024-09-16T13:17:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011136398 2024-09-16T13:17:21.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.940435+0000", 2024-09-16T13:17:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100886 2024-09-16T13:17:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.940788+0000", 2024-09-16T13:17:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035262800000000001 2024-09-16T13:17:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.945955+0000", 2024-09-16T13:17:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051675970000000003 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946370+0000", 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041468499999999998 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946409+0000", 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9254000000000001e-05 2024-09-16T13:17:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.946439+0000", 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9379000000000001e-05 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8318 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:13.981160+0000", 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.834613225, 2024-09-16T13:17:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016937001, 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8318 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981160+0000", 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981160+0000", 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981161+0000", 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.511e-06 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981170+0000", 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4120000000000001e-06 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981194+0000", 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4697000000000001e-05 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.981208+0000", 2024-09-16T13:17:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3723000000000001e-05 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.993490+0000", 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012281930999999999 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.993613+0000", 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012345800000000001 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.994131+0000", 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051729 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.997478+0000", 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033476479999999999 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.998029+0000", 2024-09-16T13:17:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00055065099999999998 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.998065+0000", 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6227999999999999e-05 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:13.998097+0000", 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1452000000000003e-05 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8369 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.7594385150000003, 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020058919000000001, 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8369 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056335+0000", 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056338+0000", 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056344+0000", 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8150000000000002e-06 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056348+0000", 2024-09-16T13:17:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:17:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.070951+0000", 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014603055 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071060+0000", 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109462 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071508+0000", 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044824200000000001 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074304+0000", 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027951719999999998 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076331+0000", 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020276529999999999 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076369+0000", 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7557e-05 2024-09-16T13:17:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076393+0000", 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4445999999999999e-05 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8370 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.759421197, 2024-09-16T13:17:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021240851000000002, 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8370 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056355+0000", 2024-09-16T13:17:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.667e-06 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056360+0000", 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6620000000000002e-06 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056366+0000", 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072672+0000", 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016306157000000002 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072746+0000", 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3648999999999999e-05 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.073364+0000", 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061885499999999997 2024-09-16T13:17:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076445+0000", 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003080396 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077482+0000", 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001036709 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077535+0000", 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3211000000000003e-05 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077592+0000", 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7664000000000002e-05 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8371 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.7593924019999996, 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021386721000000001, 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8371 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056381+0000", 2024-09-16T13:17:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056385+0000", 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056391+0000", 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6959999999999999e-06 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056396+0000", 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2549999999999999e-06 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074355+0000", 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017959098999999999 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074503+0000", 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014797200000000001 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074955+0000", 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045213300000000002 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077607+0000", 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026519500000000001 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077660+0000", 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:17:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077725+0000", 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4991000000000002e-05 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077767+0000", 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8372 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.7593713810000002, 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021557531000000001, 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056402+0000", 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1399999999999997e-07 2024-09-16T13:17:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056405+0000", 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6520000000000002e-06 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056411+0000", 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2009999999999996e-06 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056415+0000", 2024-09-16T13:17:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.211e-06 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075482+0000", 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019067163000000002 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075546+0000", 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3516000000000006e-05 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075844+0000", 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029810999999999998 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077784+0000", 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019399350000000001 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077823+0000", 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9524000000000003e-05 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077917+0000", 2024-09-16T13:17:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4090000000000002e-05 2024-09-16T13:17:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077959+0000", 2024-09-16T13:17:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1615000000000001e-05 2024-09-16T13:17:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8373 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.7593510380000001, 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022897421000000001, 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3200000000000005e-07 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056427+0000", 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-06 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056433+0000", 2024-09-16T13:17:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056437+0000", 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076470+0000", 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020032239 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076524+0000", 2024-09-16T13:17:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4475000000000003e-05 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076996+0000", 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000471905 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.078749+0000", 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001752539 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079217+0000", 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046819400000000001 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079276+0000", 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9407000000000002e-05 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079319+0000", 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8397 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085745+0000", 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.7300273710000003, 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017309516, 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8397 2024-09-16T13:17:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085745+0000", 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085745+0000", 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085746+0000", 2024-09-16T13:17:21.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085748+0000", 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.39e-06 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085754+0000", 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3569999999999997e-06 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085757+0000", 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.174e-06 2024-09-16T13:17:21.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.098475+0000", 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012718615000000001 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.098566+0000", 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1135000000000001e-05 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099043+0000", 2024-09-16T13:17:21.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000476942 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102993+0000", 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039494409999999997 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103000+0000", 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5020000000000003e-06 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103030+0000", 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0134e-05 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103055+0000", 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4468999999999999e-05 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8398 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.7299872460000003, 2024-09-16T13:17:21.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017439282, 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8398 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085788+0000", 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4719999999999998e-06 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085793+0000", 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6179999999999997e-06 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085796+0000", 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6680000000000001e-06 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099514+0000", 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013717895000000001 2024-09-16T13:17:21.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099873+0000", 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035970799999999998 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100269+0000", 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000395382 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103068+0000", 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027992680000000002 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103114+0000", 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6189000000000001e-05 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103178+0000", 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3845999999999998e-05 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103225+0000", 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6888999999999998e-05 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8399 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.7299457690000004, 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017551573000000001, 2024-09-16T13:17:21.888 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8399 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085828+0000", 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085830+0000", 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8609999999999998e-06 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085835+0000", 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.437e-06 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085837+0000", 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.571e-06 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100950+0000", 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015112537000000001 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101047+0000", 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6945000000000001e-05 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101468+0000", 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000420794 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103315+0000", 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018474769999999999 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103321+0000", 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103356+0000", 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4764000000000001e-05 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103379+0000", 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3102999999999999e-05 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.893 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8400 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.7299328679999997, 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017960752, 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8400 2024-09-16T13:17:21.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:21.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.53e-07 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085842+0000", 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2620000000000001e-06 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085847+0000", 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2119999999999997e-06 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085849+0000", 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.204e-06 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101995+0000", 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016146384999999999 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102100+0000", 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010482199999999999 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102466+0000", 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000365731 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103389+0000", 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092367199999999995 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103702+0000", 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031220400000000001 2024-09-16T13:17:21.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103754+0000", 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2318999999999998e-05 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103801+0000", 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6687999999999998e-05 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8539 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.5718013129999999, 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018486425000000001, 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8539 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0500000000000003e-07 2024-09-16T13:17:21.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243975+0000", 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800000000000002e-06 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243982+0000", 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243986+0000", 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4100000000000001e-06 2024-09-16T13:17:21.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258523+0000", 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014536604999999999 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258678+0000", 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155341 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.259218+0000", 2024-09-16T13:17:21.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053930600000000003 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262325+0000", 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031076799999999998 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262367+0000", 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1480000000000003e-05 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262417+0000", 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0555e-05 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262458+0000", 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0556999999999998e-05 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8540 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.5717818919999997, 2024-09-16T13:17:21.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018906683000000001, 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8540 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243992+0000", 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.75e-07 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243995+0000", 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1219999999999999e-06 2024-09-16T13:17:21.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244001+0000", 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5010000000000001e-06 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244006+0000", 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999996e-06 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260137+0000", 2024-09-16T13:17:21.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016131 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260200+0000", 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3782999999999996e-05 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260598+0000", 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039746600000000002 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262331+0000", 2024-09-16T13:17:21.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017327849999999999 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262786+0000", 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045546300000000003 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262851+0000", 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-05 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262898+0000", 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7098000000000002e-05 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8541 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.5717626689999999, 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019366889000000002, 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8541 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244011+0000", 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244014+0000", 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.79e-06 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244020+0000", 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6370000000000002e-06 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244025+0000", 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3359999999999997e-06 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261122+0000", 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017097161 2024-09-16T13:17:21.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261228+0000", 2024-09-16T13:17:21.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105989 2024-09-16T13:17:21.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261475+0000", 2024-09-16T13:17:21.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024757000000000002 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263022+0000", 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015466760000000001 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263300+0000", 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027828100000000001 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263344+0000", 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4008999999999998e-05 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263377+0000", 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2832e-05 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8564 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.266195+0000", 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.5495777479999999, 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017191598999999998, 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8564 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266195+0000", 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266195+0000", 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266195+0000", 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7799999999999997e-07 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266197+0000", 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1289999999999999e-06 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266201+0000", 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5710000000000002e-06 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.266205+0000", 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2429999999999999e-06 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.278604+0000", 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012398782000000001 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.278658+0000", 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3844999999999997e-05 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.278910+0000", 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025174300000000002 2024-09-16T13:17:21.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.282368+0000", 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034583880000000002 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.283290+0000", 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092164899999999997 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.283354+0000", 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4255000000000001e-05 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.283387+0000", 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2716000000000003e-05 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11776 3.1a 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.2983596620000002, 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027432719000000001, 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.923 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11776 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517415+0000", 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5239999999999999e-06 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526328+0000", 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089138059999999998 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526362+0000", 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3775000000000001e-05 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526379+0000", 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6835000000000001e-05 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526467+0000", 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7703999999999999e-05 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526653+0000", 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186379 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.527624+0000", 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097080400000000002 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543313+0000", 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015689456000000001 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544696+0000", 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013822979999999999 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544784+0000", 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8419999999999997e-05 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544846+0000", 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1717999999999994e-05 2024-09-16T13:17:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:11778 3.1 e40/23 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head v 40'57, mlcod=40'55)", 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.2918756350000002, 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01745787, 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523898+0000", 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-06 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533067+0000", 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091691120000000001 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533101+0000", 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3733999999999999e-05 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533115+0000", 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3410999999999999e-05 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533194+0000", 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9827000000000007e-05 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533223+0000", 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8937000000000001e-05 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541297+0000", 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080739850000000005 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541355+0000", 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7708999999999999e-05 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11777 3.1a 3:59cb7a3f:::rbd_object_map.112d21cb8818:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.2893707240000003, 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018649718999999999, 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.933 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11777 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:21.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526404+0000", 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.654e-06 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526409+0000", 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.882e-06 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526424+0000", 2024-09-16T13:17:21.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5328999999999998e-05 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526428+0000", 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6219999999999998e-06 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542209+0000", 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015781765999999999 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542334+0000", 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125148 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542804+0000", 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046944899999999998 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543732+0000", 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092824699999999999 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544929+0000", 2024-09-16T13:17:21.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001196522 2024-09-16T13:17:21.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545004+0000", 2024-09-16T13:17:21.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4962999999999994e-05 2024-09-16T13:17:21.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545052+0000", 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8136999999999997e-05 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11779 3.9 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.2688491419999997, 2024-09-16T13:17:21.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020806181999999999, 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11779 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546926+0000", 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9769999999999999e-06 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552330+0000", 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054043980000000004 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552353+0000", 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2875999999999999e-05 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552362+0000", 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7299999999999994e-06 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552447+0000", 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5620999999999999e-05 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552710+0000", 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026273300000000002 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.553446+0000", 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073563299999999997 2024-09-16T13:17:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.559002+0000", 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055564580000000002 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567615+0000", 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086126680000000004 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567689+0000", 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3839999999999998e-05 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567730+0000", 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1248000000000002e-05 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:21.945 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:21.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:21.947 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:21.942+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:22.048 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:22.038+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:22.054 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:22.054 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:22.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:22.149 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:22.144+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:22.227 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:22.227 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:22.227 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:22.228 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:22.228 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:22.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:22.250 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:22.244+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:22.332 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:22.332 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:22.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:22.332 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:22.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.332 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2520 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301221575, 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012988767, 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2520 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024618+0000", 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0199999999999998e-07 2024-09-16T13:17:22.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024620+0000", 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1150000000000001e-06 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024624+0000", 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.676e-06 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024626+0000", 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6730000000000001e-06 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037443+0000", 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012816869 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037522+0000", 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9060000000000005e-05 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037606+0000", 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4072000000000004e-05 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2521 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2523136~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.337 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301210203, 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013552781999999999, 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2521 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:22.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:22.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024630+0000", 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1629999999999999e-06 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024635+0000", 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5050000000000001e-06 2024-09-16T13:17:22.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024637+0000", 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4040000000000002e-06 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037635+0000", 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012997754 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037729+0000", 2024-09-16T13:17:22.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4043999999999996e-05 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038182+0000", 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045262500000000001 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2522 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3473408~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:22.342 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301198379999999, 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013780114, 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2522 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8999999999999998e-07 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024642+0000", 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3230000000000001e-06 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024646+0000", 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024648+0000", 2024-09-16T13:17:22.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4849999999999999e-06 2024-09-16T13:17:22.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038251+0000", 2024-09-16T13:17:22.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013602822000000001 2024-09-16T13:17:22.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038335+0000", 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4215999999999995e-05 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038421+0000", 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.543e-05 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.347 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2523 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301185435000001, 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014015147, 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2523 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:22.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:22.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024656+0000", 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.131e-06 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024660+0000", 2024-09-16T13:17:22.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6990000000000001e-06 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024663+0000", 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0290000000000001e-06 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038456+0000", 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013793481 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038549+0000", 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2608000000000005e-05 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038669+0000", 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011982800000000001 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2524 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301171861, 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014381392, 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2524 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024668+0000", 2024-09-16T13:17:22.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8400000000000003e-07 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024670+0000", 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.849e-06 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024675+0000", 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.451e-06 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024678+0000", 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8739999999999999e-06 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038723+0000", 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014044562999999999 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038811+0000", 2024-09-16T13:17:22.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039049+0000", 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023761100000000001 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2525 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 557056~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.357 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301137514000001, 2024-09-16T13:17:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014602618, 2024-09-16T13:17:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2525 2024-09-16T13:17:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.358 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:22.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024705+0000", 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4650000000000001e-06 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024711+0000", 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024716+0000", 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1710000000000004e-06 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039096+0000", 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014380184000000001 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039187+0000", 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1065999999999999e-05 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039304+0000", 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117333 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2526 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1507328~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301119345, 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014823846, 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.362 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2526 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024722+0000", 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024729+0000", 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024733+0000", 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039337+0000", 2024-09-16T13:17:22.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014604026000000001 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039427+0000", 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0437000000000002e-05 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039544+0000", 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116234 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.366 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2527 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1605632~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301101582000001, 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015041116, 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.367 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2527 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024738+0000", 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024746+0000", 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2130000000000008e-06 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024753+0000", 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:17:22.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024758+0000", 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7670000000000003e-06 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014820533 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039670+0000", 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1825999999999998e-05 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039779+0000", 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108424 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2528 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.371 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301076144, 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015245242000000001, 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2528 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:17:22.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024765+0000", 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1040000000000001e-06 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024772+0000", 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024776+0000", 2024-09-16T13:17:22.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039810+0000", 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015034194000000001 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039897+0000", 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6864e-05 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040008+0000", 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110802 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2529 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2555904~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301057024999999, 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015471574, 2024-09-16T13:17:22.376 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2529 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024785+0000", 2024-09-16T13:17:22.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5969999999999999e-06 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024800+0000", 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4681999999999999e-05 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024806+0000", 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040039+0000", 2024-09-16T13:17:22.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015233007999999999 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040143+0000", 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010322799999999999 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040254+0000", 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011092899999999999 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2530 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.301026745000001, 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015667606000000001, 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2530 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.381 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024813+0000", 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:22.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024815+0000", 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5160000000000001e-06 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024827+0000", 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1915e-05 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024834+0000", 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5180000000000001e-06 2024-09-16T13:17:22.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040286+0000", 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015452565999999999 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040375+0000", 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040480+0000", 2024-09-16T13:17:22.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105388 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2531 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3538944~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.300999529999999, 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015867612999999999, 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.385 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2531 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024843+0000", 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6960000000000001e-06 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024856+0000", 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-05 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024863+0000", 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6069999999999996e-06 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040513+0000", 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650662999999999 2024-09-16T13:17:22.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040599+0000", 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5439999999999995e-05 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040707+0000", 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108461 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.389 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2532 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.300969025000001, 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061912000000001, 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2532 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.390 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024871+0000", 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024874+0000", 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024881+0000", 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0570000000000004e-06 2024-09-16T13:17:22.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024886+0000", 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040742+0000", 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015856808 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040824+0000", 2024-09-16T13:17:22.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:17:22.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040932+0000", 2024-09-16T13:17:22.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108378 2024-09-16T13:17:22.394 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.394 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2533 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.300948500000001, 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016272282999999998, 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.395 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2533 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024893+0000", 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419999999999999e-06 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024900+0000", 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024903+0000", 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040965+0000", 2024-09-16T13:17:22.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061565999999999 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041052+0000", 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7213000000000007e-05 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041163+0000", 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011066899999999999 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2534 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3604480~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.300929968999998, 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016482746999999999, 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2534 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024910+0000", 2024-09-16T13:17:22.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1299999999999999e-07 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024912+0000", 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024920+0000", 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024924+0000", 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.036e-06 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041197+0000", 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016273242 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041282+0000", 2024-09-16T13:17:22.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5013999999999996e-05 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041392+0000", 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109567 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2535 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.404 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.300909307000001, 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016691902000000002, 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2535 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7599999999999997e-07 2024-09-16T13:17:22.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024932+0000", 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024949+0000", 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7053000000000001e-05 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024961+0000", 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041425+0000", 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016463851000000002 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041515+0000", 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9549999999999995e-05 2024-09-16T13:17:22.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041622+0000", 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107132 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2536 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1638400~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.300871501, 2024-09-16T13:17:22.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016880869999999999, 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2536 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024971+0000", 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.615e-06 2024-09-16T13:17:22.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024988+0000", 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6912999999999999e-05 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024993+0000", 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041653+0000", 2024-09-16T13:17:22.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016660245000000001 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041743+0000", 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0099999999999995e-05 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041848+0000", 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105404 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2537 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1671168~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.300840747999999, 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017075346000000002, 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2537 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024999+0000", 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:17:22.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025002+0000", 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025009+0000", 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025013+0000", 2024-09-16T13:17:22.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1840000000000001e-06 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041886+0000", 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016873249999999999 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041971+0000", 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4487000000000006e-05 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042074+0000", 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102847 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2538 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2621440~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.300820577, 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017314057000000001, 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2538 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025019+0000", 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025021+0000", 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025033+0000", 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1985e-05 2024-09-16T13:17:22.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025043+0000", 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3810000000000005e-06 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042107+0000", 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017064471000000001 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042203+0000", 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5810999999999995e-05 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042333+0000", 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012956099999999999 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2539 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:22.422 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.300792849, 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017500180000000001, 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2539 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025047+0000", 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025053+0000", 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025058+0000", 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025061+0000", 2024-09-16T13:17:22.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.072e-06 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042365+0000", 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017304885999999998 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042458+0000", 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2991000000000005e-05 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042546+0000", 2024-09-16T13:17:22.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7904000000000004e-05 2024-09-16T13:17:22.427 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.427 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.427 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.427 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.427 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.427 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:22.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:22.429 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:22.424+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:22.530 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:22.522+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:22.534 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:22.534 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:22.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:22.631 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:22.622+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:22.688 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:22.689 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:22.689 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:22.689 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:22.689 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:22.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:22.731 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:22.724+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:22.802 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:22.802 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:22.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:22.802 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:22.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11487 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 430080~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3092110439999995, 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014286111000000001, 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11487 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486860+0000", 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.04e-06 2024-09-16T13:17:22.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486866+0000", 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0629999999999996e-06 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486881+0000", 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5023e-05 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486889+0000", 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8149999999999995e-06 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.500906+0000", 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014016416 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501008+0000", 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102243 2024-09-16T13:17:22.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501146+0000", 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013751099999999999 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11535 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3085280709999996, 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014337727999999999, 2024-09-16T13:17:22.807 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11535 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487543+0000", 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023e-06 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487551+0000", 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0309999999999998e-06 2024-09-16T13:17:22.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487566+0000", 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5000999999999999e-05 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487578+0000", 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1066e-05 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501655+0000", 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014077151 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501747+0000", 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2670999999999994e-05 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501880+0000", 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013278500000000001 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.811 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11541 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 413696~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3084835210000003, 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014490381, 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11541 2024-09-16T13:17:22.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487588+0000", 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487592+0000", 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4920000000000004e-06 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487607+0000", 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4926000000000001e-05 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487616+0000", 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6829999999999997e-06 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501906+0000", 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0142898 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501959+0000", 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2985000000000002e-05 2024-09-16T13:17:22.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502077+0000", 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118495 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11542 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 393216~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:22.816 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3084449960000004, 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014676434, 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11542 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:22.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487626+0000", 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.034e-06 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487631+0000", 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7659999999999998e-06 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487646+0000", 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4786e-05 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487655+0000", 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6489999999999997e-06 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502110+0000", 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014455749 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502209+0000", 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8608000000000002e-05 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502302+0000", 2024-09-16T13:17:22.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2842000000000003e-05 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11544 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 466944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3084040349999997, 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014846796000000001, 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.821 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11544 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487667+0000", 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.569999999999999e-07 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487672+0000", 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2459999999999997e-06 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487686+0000", 2024-09-16T13:17:22.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4847e-05 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487695+0000", 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9109999999999999e-06 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502337+0000", 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014641861000000001 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502440+0000", 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010282599999999999 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502513+0000", 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3147999999999998e-05 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.825 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11547 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3083649079999997, 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015113645, 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11547 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487707+0000", 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.015e-06 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487711+0000", 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8289999999999998e-06 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487727+0000", 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5497000000000001e-05 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487736+0000", 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1409999999999994e-06 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502557+0000", 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014821212 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502675+0000", 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117741 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502819+0000", 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014421 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11549 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 413696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3083244860000001, 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016244525999999999, 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11549 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487747+0000", 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.094e-06 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487752+0000", 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.758e-06 2024-09-16T13:17:22.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487770+0000", 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8502999999999999e-05 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487779+0000", 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1579999999999994e-06 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502868+0000", 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015088811000000001 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502916+0000", 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8109000000000003e-05 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503990+0000", 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010740929999999999 2024-09-16T13:17:22.834 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11578 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 409600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.308274044, 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015410814, 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.835 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11578 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487798+0000", 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.646e-06 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487804+0000", 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487821+0000", 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7677999999999999e-05 2024-09-16T13:17:22.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487830+0000", 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9090000000000006e-06 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503033+0000", 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015202337999999999 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503123+0000", 2024-09-16T13:17:22.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0414000000000005e-05 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503207+0000", 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4172000000000007e-05 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11580 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.308230408, 2024-09-16T13:17:22.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015544980999999999, 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11580 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487841+0000", 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487846+0000", 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3919999999999996e-06 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487861+0000", 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4946000000000001e-05 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487869+0000", 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8750000000000006e-06 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503240+0000", 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015370511999999999 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503334+0000", 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4265000000000006e-05 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503385+0000", 2024-09-16T13:17:22.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0773999999999999e-05 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11617 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 430080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3069107100000004, 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376203000000001, 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11617 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489162+0000", 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3089999999999998e-06 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489171+0000", 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3430000000000002e-06 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489198+0000", 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6449999999999999e-05 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489216+0000", 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8057999999999999e-05 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503402+0000", 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014186196999999999 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503454+0000", 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1989e-05 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503536+0000", 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1856999999999999e-05 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.848 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11624 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 503808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3067076220000002, 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376676, 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11624 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489365+0000", 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.795e-06 2024-09-16T13:17:22.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489372+0000", 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4030000000000001e-06 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489392+0000", 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0129e-05 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489404+0000", 2024-09-16T13:17:22.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1555e-05 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503558+0000", 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014154771999999999 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503644+0000", 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5691000000000003e-05 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503740+0000", 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5330999999999999e-05 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11630 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3066542850000005, 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014893777, 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.853 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11630 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489417+0000", 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6800000000000009e-07 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489435+0000", 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7818000000000001e-05 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489450+0000", 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5072000000000001e-05 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489459+0000", 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3009999999999995e-06 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503787+0000", 2024-09-16T13:17:22.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01432719 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503889+0000", 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102604 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504310+0000", 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042082399999999998 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11632 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3066011719999997, 2024-09-16T13:17:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01524066, 2024-09-16T13:17:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11632 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489470+0000", 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.78e-07 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489475+0000", 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9400000000000001e-06 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489488+0000", 2024-09-16T13:17:22.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3176999999999999e-05 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489497+0000", 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4919999999999993e-06 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504364+0000", 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014866846 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504485+0000", 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012142999999999999 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504710+0000", 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000224897 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.862 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11639 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3065594010000003, 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015375941000000001, 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11639 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489512+0000", 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489517+0000", 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.938e-06 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489532+0000", 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.47e-05 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489540+0000", 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2949999999999994e-06 2024-09-16T13:17:22.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504765+0000", 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015225449 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504832+0000", 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6929999999999998e-05 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504887+0000", 2024-09-16T13:17:22.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4681e-05 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11641 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3065128909999997, 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015522312999999999, 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.867 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11641 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489559+0000", 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.066e-06 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489564+0000", 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9370000000000003e-06 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489579+0000", 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5466000000000001e-05 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489588+0000", 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504925+0000", 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015337353 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505025+0000", 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9770000000000002e-05 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505080+0000", 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4792e-05 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11667 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 16384~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.871 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3061521220000003, 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015353510000000001, 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11667 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:22.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489920+0000", 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.84e-06 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489928+0000", 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6969999999999993e-06 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489953+0000", 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5154999999999999e-05 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489965+0000", 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2292e-05 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505115+0000", 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015149793999999999 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505184+0000", 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8858999999999999e-05 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505272+0000", 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7873000000000004e-05 2024-09-16T13:17:22.875 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11672 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 36864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3060917170000002, 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015615053, 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.876 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11672 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:22.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489980+0000", 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489985+0000", 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490000+0000", 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490010+0000", 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.183e-06 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505335+0000", 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015325167000000001 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505471+0000", 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013581699999999999 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505594+0000", 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123168 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11676 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3060509729999996, 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015779395000000002, 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11676 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490021+0000", 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.575e-06 2024-09-16T13:17:22.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490026+0000", 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5170000000000001e-06 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490041+0000", 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4955e-05 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490049+0000", 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5520000000000005e-06 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505647+0000", 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015597489000000001 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505743+0000", 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6908000000000001e-05 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505799+0000", 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5399000000000003e-05 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11678 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3060117, 2024-09-16T13:17:22.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015992643000000001, 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11678 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490060+0000", 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3400000000000001e-06 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490065+0000", 2024-09-16T13:17:22.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4830000000000001e-06 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490079+0000", 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4854999999999999e-05 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490088+0000", 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1119999999999996e-06 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505849+0000", 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015761718000000001 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505966+0000", 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011645399999999999 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506051+0000", 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5680999999999995e-05 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.889 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11707 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 53248~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.3057925709999996, 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016106869999999999, 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:22.890 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11707 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490280+0000", 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7659999999999999e-06 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490287+0000", 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.503e-06 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490307+0000", 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9989000000000001e-05 2024-09-16T13:17:22.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490318+0000", 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0967e-05 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506171+0000", 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015853367 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506265+0000", 2024-09-16T13:17:22.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3190999999999996e-05 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506385+0000", 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120087 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:22.894 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:22.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:22.896 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:22.888+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:22.997 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:22.992+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:22.998 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:22.999 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:23.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:23.098 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:23.090+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:23.109 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:23.109 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:23.109 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:23.109 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:23.109 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:23.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:23.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:17:23.224 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:23.220+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 48.201629031000003, 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:23.234 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:17:23.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 48.201481625, 2024-09-16T13:17:23.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:17:23.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:17:23.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:17:23.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:17:23.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:23.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:17:23.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:17:23.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:23.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:17:23.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 48.201454282, 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:17:23.247 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:17:23.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:17:23.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:17:23.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:17:23.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:17:23.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:23.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.253 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.642779019999999, 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:17:23.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:23.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:23.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:17:23.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:17:23.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:17:23.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:17:23.259 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.259 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.642691427999999, 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:23.260 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:17:23.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:17:23.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.642521506000001, 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:17:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:17:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:17:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2259 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3014656~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.49119293, 2024-09-16T13:17:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016650893999999999, 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2259 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735831+0000", 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9530000000000005e-06 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735836+0000", 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7450000000000003e-06 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735839+0000", 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.915e-06 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752296+0000", 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016457241000000001 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752360+0000", 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4108000000000004e-05 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752473+0000", 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112305 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2260 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491173009000001, 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016799073000000001, 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2260 2024-09-16T13:17:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735843+0000", 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3480000000000001e-06 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735847+0000", 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5990000000000002e-06 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735849+0000", 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5859999999999999e-06 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752509+0000", 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016659377 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752559+0000", 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9774000000000002e-05 2024-09-16T13:17:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752641+0000", 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2057000000000004e-05 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2261 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491160820000001, 2024-09-16T13:17:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016960698, 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2261 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.03e-07 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735855+0000", 2024-09-16T13:17:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735860+0000", 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1110000000000001e-06 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735862+0000", 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752695+0000", 2024-09-16T13:17:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01683343 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752744+0000", 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8414000000000001e-05 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752814+0000", 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0623000000000005e-05 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2262 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1048576~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491150206, 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017224107999999998, 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2262 2024-09-16T13:17:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735865+0000", 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.91e-07 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735866+0000", 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1480000000000001e-06 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735870+0000", 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735877+0000", 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4370000000000001e-06 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752871+0000", 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016994061000000001 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752962+0000", 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0470000000000006e-05 2024-09-16T13:17:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753089+0000", 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126642 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2263 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491134747, 2024-09-16T13:17:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017392088999999999, 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2263 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-07 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735881+0000", 2024-09-16T13:17:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735885+0000", 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4180000000000001e-06 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735887+0000", 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-06 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753116+0000", 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017228963 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753184+0000", 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7374999999999998e-05 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753272+0000", 2024-09-16T13:17:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8323000000000001e-05 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2264 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2064384~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491124895999999, 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017630541999999999, 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2264 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735891+0000", 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735895+0000", 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735897+0000", 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4669999999999998e-06 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753316+0000", 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017418685 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753403+0000", 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7590000000000007e-05 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753520+0000", 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116863 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2265 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1081344~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491115014000002, 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017739263000000002, 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2265 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735901+0000", 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735904+0000", 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.304e-06 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735907+0000", 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3599999999999999e-06 2024-09-16T13:17:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753534+0000", 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017627446000000001 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753576+0000", 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1474999999999998e-05 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753639+0000", 2024-09-16T13:17:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3188999999999994e-05 2024-09-16T13:17:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2266 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2097152~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491105781000002, 2024-09-16T13:17:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017970893000000002, 2024-09-16T13:17:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2266 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735914+0000", 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1699999999999996e-06 2024-09-16T13:17:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735918+0000", 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6859999999999999e-06 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735920+0000", 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4870000000000001e-06 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753685+0000", 2024-09-16T13:17:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017764582000000001 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753771+0000", 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6259000000000002e-05 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753880+0000", 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010844300000000001 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2267 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491091879999999, 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018110804000000001, 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2267 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735924+0000", 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735928+0000", 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.416e-06 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735930+0000", 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753898+0000", 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017967852999999999 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753955+0000", 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6662000000000001e-05 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754034+0000", 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8904000000000002e-05 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2268 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3080192~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491082394999999, 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018252866, 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2268 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735934+0000", 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0950000000000001e-06 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735938+0000", 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735941+0000", 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8750000000000002e-06 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754071+0000", 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018129941 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754120+0000", 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8359000000000002e-05 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754185+0000", 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5358999999999995e-05 2024-09-16T13:17:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2269 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491071015999999, 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018533516, 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2269 2024-09-16T13:17:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735945+0000", 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1149999999999999e-06 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735948+0000", 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5250000000000001e-06 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735951+0000", 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9909999999999998e-06 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754270+0000", 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018318081999999999 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754325+0000", 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5648e-05 2024-09-16T13:17:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754477+0000", 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015188199999999999 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2270 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1114112~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491060547, 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019103282999999999, 2024-09-16T13:17:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2270 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735955+0000", 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735956+0000", 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.096e-06 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735959+0000", 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5140000000000002e-06 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735963+0000", 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.053e-06 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754822+0000", 2024-09-16T13:17:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018858883 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754941+0000", 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118789 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755057+0000", 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116272 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2271 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1146880~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491038735, 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019332309999999998, 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2271 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3500000000000002e-07 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735982+0000", 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735987+0000", 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1930000000000003e-06 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735993+0000", 2024-09-16T13:17:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4659999999999996e-06 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755081+0000", 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019088187 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755194+0000", 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011223299999999999 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755308+0000", 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114663 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2272 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 163840~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.491016904999999, 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019476291, 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2272 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736003+0000", 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2279999999999998e-06 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736007+0000", 2024-09-16T13:17:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.506e-06 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736011+0000", 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9280000000000003e-06 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755336+0000", 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019325189999999999 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755393+0000", 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7293999999999997e-05 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755474+0000", 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0635000000000003e-05 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:23.338 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:23.339 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone9 from 0xaa282b8 (overlap 0x5065db) to 0x2028be 2024-09-16T13:17:23.340 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:23.330+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:23.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:23.415 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #8, image image_client.0-clone8 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent9 2024-09-16T13:17:23.438 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:23.438 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:23.441 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:23.430+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:23.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:23.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:23.544 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:23.540+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:23.553 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:23.553 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:23.553 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:23.553 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:23.553 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:23.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:23.642 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:23.634+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:23.655 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:23.661 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:23.661 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:23.670 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:23.683 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:23.702 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:23.721 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:23.742 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:23.742 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:23.736+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:23.765 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:23.783 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:23.784 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:23.788 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:23.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:23.812 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:23.814 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:23.815 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:23.872 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:23.868+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:23.893 INFO:teuthology.orchestra.run.smithi098.stdout:1782 write 0xb37c304 thru 0xb388948 (0xc645 bytes) 2024-09-16T13:17:23.899 INFO:teuthology.orchestra.run.smithi098.stdout:1783 read 0x9df07fc thru 0x9dfc11d (0xb922 bytes) 2024-09-16T13:17:23.899 INFO:teuthology.orchestra.run.smithi098.stdout:1784 read 0x2744a5 thru 0x278da6 (0x4902 bytes) 2024-09-16T13:17:23.900 INFO:teuthology.orchestra.run.smithi098.stdout:1785 write 0x2e3119 thru 0x2f180d (0xe6f5 bytes) 2024-09-16T13:17:23.909 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:23.909 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:23.914 INFO:teuthology.orchestra.run.smithi098.stdout:1787 writesame 0x851dbad thru 0x8529c7d data_size 0x1(0xc0d1 bytes) 2024-09-16T13:17:23.918 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:23.944 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:23.934+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:23.949 INFO:teuthology.orchestra.run.smithi098.stdout:1788 write 0x9f63362 thru 0x9f69978 (0x6617 bytes) 2024-09-16T13:17:23.957 INFO:teuthology.orchestra.run.smithi098.stdout:1789 write 0x1c9fcc5 thru 0x1ca992b (0x9c67 bytes) 2024-09-16T13:17:23.970 INFO:teuthology.orchestra.run.smithi098.stdout:1792 write 0x918202d thru 0x9188165 (0x6139 bytes) 2024-09-16T13:17:23.976 INFO:teuthology.orchestra.run.smithi098.stdout:1794 read 0x9184c42 thru 0x91900f7 (0xb4b6 bytes) 2024-09-16T13:17:23.977 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:23.977 INFO:teuthology.orchestra.run.smithi098.stdout:1796 read 0x4374364 thru 0x4381296 (0xcf33 bytes) 2024-09-16T13:17:23.977 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:23.979 INFO:teuthology.orchestra.run.smithi098.stdout:1798 trunc from 0xb388949 to 0xa136734 2024-09-16T13:17:24.002 INFO:teuthology.orchestra.run.smithi098.stdout:1799 punch from 0x3296ca4 to 0x329ebf7, (0x7f53 bytes) 2024-09-16T13:17:24.018 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:24.018 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:24.021 INFO:teuthology.orchestra.run.smithi098.stdout:1800 write 0xd76a47d thru 0xd778afd (0xe681 bytes) 2024-09-16T13:17:24.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:24.043 INFO:teuthology.orchestra.run.smithi098.stdout:1802 trunc from 0xd778afe to 0x276cbbf 2024-09-16T13:17:24.045 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:24.038+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:24.085 INFO:teuthology.orchestra.run.smithi098.stdout:1803 read 0x1356311 thru 0x135f86d (0x955d bytes) 2024-09-16T13:17:24.085 INFO:teuthology.orchestra.run.smithi098.stdout:1805 read 0x820da5 thru 0x82635b (0x55b7 bytes) 2024-09-16T13:17:24.089 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:24.106 INFO:teuthology.orchestra.run.smithi098.stdout:1808 writesame 0x881470b thru 0x8824559 data_size 0x1(0xfe4f bytes) 2024-09-16T13:17:24.106 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:24.106 INFO:teuthology.orchestra.run.smithi098.stdout:1810 write 0x75161ce thru 0x7524d7f (0xebb2 bytes) 2024-09-16T13:17:24.109 INFO:teuthology.orchestra.run.smithi098.stdout:1811 read 0x864d899 thru 0x8658f3c (0xb6a4 bytes) 2024-09-16T13:17:24.109 INFO:teuthology.orchestra.run.smithi098.stdout:1812 write 0xe76d50 thru 0xe7e8e6 (0x7b97 bytes) 2024-09-16T13:17:24.112 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:24.112 INFO:teuthology.orchestra.run.smithi098.stdout:1815 read 0x3de004b thru 0x3de0162 (0x118 bytes) 2024-09-16T13:17:24.113 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:24.113 INFO:teuthology.orchestra.run.smithi098.stdout:1819 read 0x3030359 thru 0x303704d (0x6cf5 bytes) 2024-09-16T13:17:24.114 INFO:teuthology.orchestra.run.smithi098.stdout:1820 punch from 0x4ea93c7 to 0x4eac479, (0x30b2 bytes) 2024-09-16T13:17:24.114 INFO:teuthology.orchestra.run.smithi098.stdout:1821 read 0xe22f74 thru 0xe31214 (0xe2a1 bytes) 2024-09-16T13:17:24.115 INFO:teuthology.orchestra.run.smithi098.stdout:1824 read 0x1322379 thru 0x13295b0 (0x7238 bytes) 2024-09-16T13:17:24.115 INFO:teuthology.orchestra.run.smithi098.stdout:1825 read 0x25ef415 thru 0x25ef4bd (0xa9 bytes) 2024-09-16T13:17:24.116 INFO:teuthology.orchestra.run.smithi098.stdout:1826 read 0x4ff2dc0 thru 0x4ffce23 (0xa064 bytes) 2024-09-16T13:17:24.126 INFO:teuthology.orchestra.run.smithi098.stdout:1827 trunc from 0x882455a to 0xdaf6432 2024-09-16T13:17:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:1829 write 0x8123347 thru 0x812f793 (0xc44d bytes) 2024-09-16T13:17:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:24.139 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:24.146 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:24.138+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:24.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:24.155 INFO:teuthology.orchestra.run.smithi098.stdout:1830 trunc from 0xdaf6432 to 0x8af8dad 2024-09-16T13:17:24.170 INFO:teuthology.orchestra.run.smithi098.stdout:1831 write 0x2b0167b thru 0x2b0ac38 (0x95be bytes) 2024-09-16T13:17:24.174 INFO:teuthology.orchestra.run.smithi098.stdout:1832 write 0x2907235 thru 0x290de92 (0x6c5e bytes) 2024-09-16T13:17:24.189 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:17:24.194 INFO:teuthology.orchestra.run.smithi098.stdout:1833 trunc from 0x8af8dad to 0x2502c0 2024-09-16T13:17:24.247 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:24.238+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:24.252 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:24.252 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:24.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:24.347 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:24.340+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:24.359 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:24.359 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:24.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:24.448 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:24.442+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:24.469 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:24.469 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:24.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:24.550 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:24.544+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:24.576 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:24.576 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:24.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:24.651 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:24.646+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:24.733 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:24.733 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:24.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:24.752 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:24.748+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:24.853 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:24.848+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:24.854 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:24.854 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:24.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:24.954 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:24.948+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:24.966 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:24.967 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:24.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:25.055 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:25.048+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:25.070 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:25.070 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:25.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:25.156 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:25.146+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:25.181 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:25.181 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:25.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:25.245 INFO:teuthology.orchestra.run.smithi098.stdout:1834 writesame 0xd923ca5 thru 0xd92c26a data_size 0x1(0x85c6 bytes) 2024-09-16T13:17:25.245 INFO:teuthology.orchestra.run.smithi098.stdout:1835 read 0x51ce3a4 thru 0x51cfc2f (0x188c bytes) 2024-09-16T13:17:25.249 INFO:teuthology.orchestra.run.smithi098.stdout:1836 read 0xb22328b thru 0xb22ebc4 (0xb93a bytes) 2024-09-16T13:17:25.249 INFO:teuthology.orchestra.run.smithi098.stdout:1838 write 0x48d7239 thru 0x48dd0b6 (0x5e7e bytes) 2024-09-16T13:17:25.253 INFO:teuthology.orchestra.run.smithi098.stdout:1839 read 0x2c39ef9 thru 0x2c408f0 (0x69f8 bytes) 2024-09-16T13:17:25.255 INFO:teuthology.orchestra.run.smithi098.stdout:1840 trunc from 0xd92c26b to 0xc8b9d7f 2024-09-16T13:17:25.256 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:25.252+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:25.268 INFO:teuthology.orchestra.run.smithi098.stdout:1842 read 0x49ca2ce thru 0x49da234 (0xff67 bytes) 2024-09-16T13:17:25.269 INFO:teuthology.orchestra.run.smithi098.stdout:1843 write 0x1f2c803 thru 0x1f2ee65 (0x2663 bytes) 2024-09-16T13:17:25.276 INFO:teuthology.orchestra.run.smithi098.stdout:1844 clone 11 order 22 su 131072 sc 10 2024-09-16T13:17:25.335 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:25.336 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:25.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:25.357 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:25.350+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:25.442 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:25.442 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:25.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:25.458 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:25.452+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:25.560 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:25.556+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:25.598 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:25.599 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:25.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:25.660 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:25.650+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:25.706 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:25.706 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:25.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:25.761 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:25.756+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:25.815 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:25.815 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:25.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:25.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:25.872 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:25.868+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:25.929 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:25.929 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:25.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:25.964 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:25.954+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:25.974 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:25.996 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:26.036 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:26.036 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:26.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:26.065 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:26.060+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:26.166 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:26.160+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:26.178 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:26.178 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:26.178 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-16T13:17:26.178 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:17:26.179 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:17:26.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-16T13:17:26.195 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:26.195 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:26.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:17:26.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:26.237 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:26.266 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:26.260+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:26.271 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:26.299 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:26.301 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:26.301 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T13:17:26.301 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:26.321 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:26.333 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone10 from 0xc8b9d7f (overlap 0x2502c0) to 0x1634d9 2024-09-16T13:17:26.337 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:26.337 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:26.337 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:26.338 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:26.345 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:26.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:26.367 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:26.358+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:26.368 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:26.448 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:26.448 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:26.448 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:26.448 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:26.449 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:26.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:26.460 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #9, image image_client.0-clone9 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent10 2024-09-16T13:17:26.468 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:26.458+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:26.472 INFO:teuthology.orchestra.run.smithi098.stdout:1845 read 0xacf7801 thru 0xad07363 (0xfb63 bytes) 2024-09-16T13:17:26.473 INFO:teuthology.orchestra.run.smithi098.stdout:1847 write 0x55639b2 thru 0x556e728 (0xad77 bytes) 2024-09-16T13:17:26.490 INFO:teuthology.orchestra.run.smithi098.stdout:1848 write 0x7603bd5 thru 0x760a54a (0x6976 bytes) 2024-09-16T13:17:26.498 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:26.499 INFO:teuthology.orchestra.run.smithi098.stdout:1850 write 0x7170bf thru 0x71adf4 (0x3d36 bytes) 2024-09-16T13:17:26.509 INFO:teuthology.orchestra.run.smithi098.stdout:1851 read 0x6b8051 thru 0x6bf5bc (0x756c bytes) 2024-09-16T13:17:26.509 INFO:teuthology.orchestra.run.smithi098.stdout:1852 read 0xbc4a1c5 thru 0xbc4b617 (0x1453 bytes) 2024-09-16T13:17:26.509 INFO:teuthology.orchestra.run.smithi098.stdout:1853 writesame 0x37e0f70 thru 0x37e72e8 data_size 0xb(0x6379 bytes) 2024-09-16T13:17:26.519 INFO:teuthology.orchestra.run.smithi098.stdout:1854 write 0x8b03cf4 thru 0x8b08c3d (0x4f4a bytes) 2024-09-16T13:17:26.527 INFO:teuthology.orchestra.run.smithi098.stdout:1859 punch from 0x587f09 to 0x5967c1, (0xe8b8 bytes) 2024-09-16T13:17:26.527 INFO:teuthology.orchestra.run.smithi098.stdout:1860 punch from 0x46b5bd1 to 0x46b85c9, (0x29f8 bytes) 2024-09-16T13:17:26.527 INFO:teuthology.orchestra.run.smithi098.stdout:1861 writesame 0x9d75fba thru 0x9d7b010 data_size 0x1(0x5057 bytes) 2024-09-16T13:17:26.552 INFO:teuthology.orchestra.run.smithi098.stdout:1862 read 0x1880ad4 thru 0x188b66e (0xab9b bytes) 2024-09-16T13:17:26.555 INFO:teuthology.orchestra.run.smithi098.stdout:1863 read 0x73db739 thru 0x73df89e (0x4166 bytes) 2024-09-16T13:17:26.555 INFO:teuthology.orchestra.run.smithi098.stdout:1864 punch from 0x96527e8 to 0x9657623, (0x4e3b bytes) 2024-09-16T13:17:26.555 INFO:teuthology.orchestra.run.smithi098.stdout:1865 read 0x3e34cd thru 0x3ef884 (0xc3b8 bytes) 2024-09-16T13:17:26.556 INFO:teuthology.orchestra.run.smithi098.stdout:1867 read 0x68c81cc thru 0x68ca109 (0x1f3e bytes) 2024-09-16T13:17:26.556 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:26.556 INFO:teuthology.orchestra.run.smithi098.stdout:1869 punch from 0xa286461 to 0xa2866b7, (0x256 bytes) 2024-09-16T13:17:26.563 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11500 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 356352~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0601475889999996, 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014537983000000001, 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11500 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496474+0000", 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.801e-06 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496480+0000", 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1140000000000002e-06 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496484+0000", 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8770000000000003e-06 2024-09-16T13:17:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507573+0000", 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011088473 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507624+0000", 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1591999999999997e-05 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511010+0000", 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033857380000000001 2024-09-16T13:17:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11501 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 430080~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0601312220000008, 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014726846, 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11501 2024-09-16T13:17:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496489+0000", 2024-09-16T13:17:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3700000000000001e-07 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496491+0000", 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9570000000000001e-06 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496497+0000", 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2550000000000003e-06 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496501+0000", 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8639999999999998e-06 2024-09-16T13:17:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511060+0000", 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014559462 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511125+0000", 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4399999999999993e-05 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511215+0000", 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0470999999999995e-05 2024-09-16T13:17:26.571 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11530 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 339968~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0591970340000003, 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014141661, 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11530 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497423+0000", 2024-09-16T13:17:26.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-07 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497434+0000", 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1195e-05 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497442+0000", 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.323e-06 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497446+0000", 2024-09-16T13:17:26.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1049999999999997e-06 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511275+0000", 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013828872000000001 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511389+0000", 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114493 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511564+0000", 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017508700000000001 2024-09-16T13:17:26.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11533 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 413696~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0591679500000009, 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014296237999999999, 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.576 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11533 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.51e-07 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497458+0000", 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9139999999999996e-06 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497464+0000", 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0469999999999999e-06 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497467+0000", 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5030000000000002e-06 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511593+0000", 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014125692 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511669+0000", 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5950000000000003e-05 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511748+0000", 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8681000000000001e-05 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11551 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 335872~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0591119229999997, 2024-09-16T13:17:26.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014415314, 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11551 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497510+0000", 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.012e-06 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497516+0000", 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7880000000000003e-06 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497524+0000", 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9049999999999997e-06 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511769+0000", 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245809 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511818+0000", 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8092e-05 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511923+0000", 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010532299999999999 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.584 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11553 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 413696~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0590890739999992, 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014561233999999999, 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11553 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497531+0000", 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1899999999999998e-07 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497533+0000", 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9149999999999999e-06 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497539+0000", 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.088e-06 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497550+0000", 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0834e-05 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511962+0000", 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411947 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512018+0000", 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6755999999999997e-05 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512092+0000", 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3275000000000006e-05 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.588 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.589 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.589 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11576 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 335872~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.589 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499621+0000", 2024-09-16T13:17:26.589 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0569980979999993, 2024-09-16T13:17:26.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013787382000000001, 2024-09-16T13:17:26.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.589 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.590 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.590 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11576 2024-09-16T13:17:26.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.590 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499621+0000", 2024-09-16T13:17:26.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499621+0000", 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499622+0000", 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6400000000000003e-07 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499624+0000", 2024-09-16T13:17:26.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499630+0000", 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9429999999999997e-06 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499634+0000", 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513288+0000", 2024-09-16T13:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013654463 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513342+0000", 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4357000000000002e-05 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513409+0000", 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6456000000000002e-05 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11577 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 409600~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0569818299999998, 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013920979999999999, 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11577 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499640+0000", 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1430000000000001e-06 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499646+0000", 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1029999999999998e-06 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499650+0000", 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6370000000000001e-06 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513442+0000", 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013791805000000001 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513494+0000", 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2074999999999999e-05 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513559+0000", 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4874000000000003e-05 2024-09-16T13:17:26.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11628 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 471040~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0566246360000005, 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013902296999999999, 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11628 2024-09-16T13:17:26.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3599999999999999e-07 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499997+0000", 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.181e-06 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500004+0000", 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4609999999999996e-06 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500008+0000", 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0629999999999999e-06 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513576+0000", 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013567818000000001 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513626+0000", 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0013999999999999e-05 2024-09-16T13:17:26.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513897+0000", 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027132399999999998 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11637 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0560893989999993, 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013965546000000001, 2024-09-16T13:17:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11637 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500531+0000", 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2699999999999999e-07 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500534+0000", 2024-09-16T13:17:26.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.055e-06 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500542+0000", 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6389999999999998e-06 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500553+0000", 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0974e-05 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514249+0000", 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013696086 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514308+0000", 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8269000000000001e-05 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514496+0000", 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018799600000000001 2024-09-16T13:17:26.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11638 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0560599170000007, 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014058684, 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.607 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11638 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500563+0000", 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0790000000000001e-06 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500580+0000", 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6472000000000001e-05 2024-09-16T13:17:26.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500586+0000", 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2199999999999997e-06 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514522+0000", 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013935774 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514564+0000", 2024-09-16T13:17:26.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2549000000000002e-05 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514618+0000", 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3942e-05 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11642 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.055975084, 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014093902, 2024-09-16T13:17:26.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11642 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500645+0000", 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5200000000000002e-07 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500656+0000", 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1372999999999999e-05 2024-09-16T13:17:26.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500664+0000", 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9019999999999999e-06 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500677+0000", 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2954e-05 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514649+0000", 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013971417 2024-09-16T13:17:26.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514692+0000", 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3154000000000001e-05 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514738+0000", 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6650000000000002e-05 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11646 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0559368219999996, 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245793, 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11646 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500685+0000", 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2019999999999998e-06 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500698+0000", 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.279e-05 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500702+0000", 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7629999999999998e-06 2024-09-16T13:17:26.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514772+0000", 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014070384999999999 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514865+0000", 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2980999999999996e-05 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514928+0000", 2024-09-16T13:17:26.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3114000000000006e-05 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11648 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0559136920000007, 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411604999999999, 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.620 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11648 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9900000000000001e-07 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500713+0000", 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.793e-06 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500719+0000", 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500722+0000", 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3809999999999999e-06 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514973+0000", 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014250924 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515031+0000", 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7864e-05 2024-09-16T13:17:26.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515117+0000", 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6697e-05 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11654 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0555468579999996, 2024-09-16T13:17:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014438998, 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11654 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8699999999999995e-07 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501075+0000", 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501082+0000", 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.844e-06 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501086+0000", 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7749999999999999e-06 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515152+0000", 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014065891000000001 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515232+0000", 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0461000000000001e-05 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515512+0000", 2024-09-16T13:17:26.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027934599999999998 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11675 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0550842639999995, 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014239586, 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.629 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11675 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:26.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501536+0000", 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501539+0000", 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3529999999999999e-06 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501546+0000", 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.109e-06 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501551+0000", 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5109999999999997e-06 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515586+0000", 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014035334999999999 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515689+0000", 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010244200000000001 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515775+0000", 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6150999999999996e-05 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11679 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:26.633 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0550343400000006, 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014462922, 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11679 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501586+0000", 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501588+0000", 2024-09-16T13:17:26.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3760000000000002e-06 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501594+0000", 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9950000000000002e-06 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501598+0000", 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7239999999999998e-06 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515840+0000", 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014242546999999999 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515949+0000", 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108591 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516048+0000", 2024-09-16T13:17:26.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9123000000000006e-05 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11686 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0550177680000008, 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014648357000000001, 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.638 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11686 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501604+0000", 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.846e-06 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501609+0000", 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3630000000000002e-06 2024-09-16T13:17:26.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501613+0000", 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3749999999999999e-06 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516077+0000", 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014464351 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516173+0000", 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5742000000000006e-05 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516250+0000", 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7312999999999997e-05 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11689 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0550032040000001, 2024-09-16T13:17:26.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014770922000000001, 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11689 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501617+0000", 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501618+0000", 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6780000000000001e-06 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501624+0000", 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3750000000000002e-06 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501627+0000", 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5149999999999998e-06 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516279+0000", 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014651928999999999 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.645 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516336+0000", 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6724000000000001e-05 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516387+0000", 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1334e-05 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11778 3.1 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:26.646 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0376690760000002, 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022495198000000001, 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11778 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:26.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518952+0000", 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.711e-06 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522261+0000", 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033087429999999998 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522296+0000", 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5335e-05 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522306+0000", 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0074999999999999e-05 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522379+0000", 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2554999999999999e-05 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522572+0000", 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019323100000000001 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.523266+0000", 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069379000000000001 2024-09-16T13:17:26.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.534086+0000", 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010820112999999999 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541312+0000", 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072264440000000003 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541394+0000", 2024-09-16T13:17:26.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1710000000000002e-05 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541446+0000", 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1490999999999999e-05 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:26.652 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:26.653 INFO:teuthology.orchestra.run.smithi098.stdout:1870 writesame 0xca07304 thru 0xca1615c data_size 0x2b(0xee59 bytes) 2024-09-16T13:17:26.653 INFO:teuthology.orchestra.run.smithi098.stdout:1871 writesame 0x886a50b thru 0x88706c2 data_size 0x8(0x61b8 bytes) 2024-09-16T13:17:26.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:26.655 INFO:teuthology.orchestra.run.smithi098.stdout:1872 read 0x87cfc79 thru 0x87d6b2c (0x6eb4 bytes) 2024-09-16T13:17:26.655 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:26.655 INFO:teuthology.orchestra.run.smithi098.stdout:1876 punch from 0xcf01db to 0xcfa36c, (0xa191 bytes) 2024-09-16T13:17:26.655 INFO:teuthology.orchestra.run.smithi098.stdout:1878 writesame 0xcd75c57 thru 0xcd7eb09 data_size 0x63(0x8eb3 bytes) 2024-09-16T13:17:26.655 INFO:teuthology.orchestra.run.smithi098.stdout:1879 trunc from 0xcd7eb0a to 0x165df64 2024-09-16T13:17:26.656 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:26.646+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:26.697 INFO:teuthology.orchestra.run.smithi098.stdout:1880 writesame 0xe69eef4 thru 0xe6a4921 data_size 0x7(0x5a2e bytes) 2024-09-16T13:17:26.697 INFO:teuthology.orchestra.run.smithi098.stdout:1882 punch from 0x69cd304 to 0x69d9024, (0xbd20 bytes) 2024-09-16T13:17:26.698 INFO:teuthology.orchestra.run.smithi098.stdout:1883 read 0xda062b1 thru 0xda124ef (0xc23f bytes) 2024-09-16T13:17:26.698 INFO:teuthology.orchestra.run.smithi098.stdout:1885 clone 12 order 25 su 4194304 sc 2 2024-09-16T13:17:26.747 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:26.747 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:26.755 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:26.750+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:26.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:26.856 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:26.852+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:26.900 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:26.900 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:26.900 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:26.900 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:26.900 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:26.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:26.957 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:26.946+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:27.009 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:27.009 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:27.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:27.009 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:27.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.010 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6966 3.0 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:27.010 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:27.010 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.448272729999999, 2024-09-16T13:17:27.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095509229999999994, 2024-09-16T13:17:27.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.010 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.010 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6966 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554532+0000", 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2000000000000001e-07 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554534+0000", 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9850000000000001e-06 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554537+0000", 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.664e-06 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554541+0000", 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2909999999999999e-06 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557170+0000", 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026295310000000001 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557345+0000", 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017473200000000001 2024-09-16T13:17:27.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557618+0000", 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027282800000000002 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563504+0000", 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058862649999999999 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563923+0000", 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000419004 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564003+0000", 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0072000000000002e-05 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564082+0000", 2024-09-16T13:17:27.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9231000000000001e-05 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11223 3.8 e40/23 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head v 40'368, mlcod=39'365)", 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.5361016460000005, 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01312405, 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466704+0000", 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.792e-06 2024-09-16T13:17:27.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471144+0000", 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044394830000000001 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471180+0000", 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5884999999999999e-05 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471194+0000", 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4258e-05 2024-09-16T13:17:27.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471273+0000", 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9159999999999994e-05 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471305+0000", 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2236000000000001e-05 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479753+0000", 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0084474049999999998 2024-09-16T13:17:27.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479826+0000", 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3831000000000005e-05 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11776 3.1a e40/23 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head v 40'72, mlcod=40'70)", 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.4752048490000007, 2024-09-16T13:17:27.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595333, 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527601+0000", 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.522e-06 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536775+0000", 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091746569999999993 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536818+0000", 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2435000000000003e-05 2024-09-16T13:17:27.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536828+0000", 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0624e-05 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536931+0000", 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010280000000000001 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536961+0000", 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0060999999999999e-05 2024-09-16T13:17:27.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544146+0000", 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071846339999999996 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544195+0000", 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8600000000000002e-05 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13719 3.19 3:99fbe691:::rbd_data.112df1cc59bd.000000000000000c:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.7220352779999999, 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011713708999999999, 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13719 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.025 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280771+0000", 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9240000000000001e-06 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280782+0000", 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1642e-05 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280806+0000", 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4119000000000001e-05 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280815+0000", 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7150000000000004e-06 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012396900000000001 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281142+0000", 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000203298 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:27.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281540+0000", 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039763500000000002 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.286598+0000", 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050579090000000002 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292380+0000", 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057823390000000001 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292433+0000", 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2612999999999999e-05 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292482+0000", 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9546000000000003e-05 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13720 3.19 3:99acede8:::rbd_data.112df1cc59bd.000000000000000d:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.7219720860000001, 2024-09-16T13:17:27.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011830879000000001, 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13720 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280836+0000", 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.703e-06 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280844+0000", 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280849+0000", 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.254e-06 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282739+0000", 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001889672 2024-09-16T13:17:27.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282875+0000", 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013641 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.283217+0000", 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00034147099999999998 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.288528+0000", 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053110140000000002 2024-09-16T13:17:27.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292558+0000", 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040304429999999999 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292620+0000", 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1588999999999996e-05 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292663+0000", 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2922e-05 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13726 3.19 3:9e45447e:::rbd_data.112df1cc59bd.0000000000000034:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.7218713729999999, 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012171434, 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.036 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13726 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280934+0000", 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9299999999999996e-07 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:17:27.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280950+0000", 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1141000000000001e-05 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280955+0000", 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5560000000000001e-06 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284230+0000", 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032752789999999999 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284286+0000", 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6456999999999998e-05 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284430+0000", 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014382899999999999 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.291179+0000", 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067490099999999997 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293014+0000", 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018347719999999999 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293069+0000", 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5325999999999998e-05 2024-09-16T13:17:27.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293104+0000", 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4892999999999999e-05 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14647 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 3801088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36767343600000002, 2024-09-16T13:17:27.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0094151970000000001, 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14647 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635132+0000", 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4300000000000001e-07 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635136+0000", 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3660000000000002e-06 2024-09-16T13:17:27.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635150+0000", 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4491999999999999e-05 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635159+0000", 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2630000000000003e-06 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644405+0000", 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0092462220000000001 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644476+0000", 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0958999999999996e-05 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644546+0000", 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9951999999999997e-05 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.046 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14648 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4325376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36763574900000001, 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096264239999999997, 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.047 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14648 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:27.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635169+0000", 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.071e-06 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635173+0000", 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635188+0000", 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4389000000000001e-05 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635198+0000", 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0566e-05 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644579+0000", 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093805399999999997 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644670+0000", 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1240000000000001e-05 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644795+0000", 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012454 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14662 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1703936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.051 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36739018299999998, 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096453119999999996, 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14662 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635415+0000", 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5e-06 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635421+0000", 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730000000000001e-06 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635436+0000", 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4909000000000001e-05 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635450+0000", 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3183999999999999e-05 2024-09-16T13:17:27.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644833+0000", 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093835959999999993 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644914+0000", 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0624000000000006e-05 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645059+0000", 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014542600000000001 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14663 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2228224~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36734502099999999, 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098295339999999991, 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.056 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14663 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:27.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635460+0000", 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1500000000000004e-07 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635463+0000", 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.478e-06 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635472+0000", 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4230000000000005e-06 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635498+0000", 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6543e-05 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645115+0000", 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096171710000000007 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645193+0000", 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7245999999999999e-05 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645289+0000", 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5958000000000005e-05 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14664 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:27.060 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36729853099999998, 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010033297, 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14664 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635506+0000", 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5099999999999998e-07 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635511+0000", 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7820000000000001e-06 2024-09-16T13:17:27.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635520+0000", 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5969999999999992e-06 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635529+0000", 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3249999999999997e-06 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645337+0000", 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098082589999999997 2024-09-16T13:17:27.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645427+0000", 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9922999999999999e-05 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645539+0000", 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011156 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14665 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36726930299999999, 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010434005999999999, 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14665 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635538+0000", 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8329999999999998e-06 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635546+0000", 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0760000000000003e-06 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635551+0000", 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6160000000000004e-06 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645677+0000", 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010126546 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645821+0000", 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014406799999999999 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645969+0000", 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014730900000000001 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.068 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14678 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 0~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36700985899999999, 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012023611999999999, 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14678 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635797+0000", 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7039999999999999e-06 2024-09-16T13:17:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635810+0000", 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3343000000000001e-05 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635845+0000", 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4751e-05 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635863+0000", 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8210999999999999e-05 2024-09-16T13:17:27.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646005+0000", 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010141766999999999 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646070+0000", 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5324e-05 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647818+0000", 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017475119999999999 2024-09-16T13:17:27.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14679 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1835008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36692215299999997, 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012069985, 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14679 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635883+0000", 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3239999999999999e-06 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635889+0000", 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635905+0000", 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6022999999999999e-05 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635911+0000", 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3740000000000003e-06 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647849+0000", 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011937906999999999 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647895+0000", 2024-09-16T13:17:27.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5507999999999999e-05 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647952+0000", 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7132000000000001e-05 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14680 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2359296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36688587099999997, 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012172090999999999, 2024-09-16T13:17:27.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14680 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635919+0000", 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8299999999999995e-07 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635922+0000", 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.027e-06 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635931+0000", 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6230000000000003e-06 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635936+0000", 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0069999999999997e-06 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647981+0000", 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012045273 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648033+0000", 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648090+0000", 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7154000000000002e-05 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14681 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4718592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.081 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36686287400000001, 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012312528, 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14681 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:27.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635942+0000", 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7599999999999996e-07 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635959+0000", 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6954999999999999e-05 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635968+0000", 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8929999999999994e-06 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635973+0000", 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0170000000000004e-06 2024-09-16T13:17:27.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648120+0000", 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012146847000000001 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648170+0000", 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0368000000000003e-05 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648254+0000", 2024-09-16T13:17:27.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3671999999999994e-05 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14694 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36682579999999998, 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012792843999999999, 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.086 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14694 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635979+0000", 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0699999999999996e-07 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635982+0000", 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8200000000000001e-06 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635990+0000", 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0239999999999997e-06 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635995+0000", 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648296+0000", 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0123009 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648397+0000", 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101042 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648771+0000", 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037435099999999998 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14695 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1966080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36680378200000002, 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012992136, 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.090 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.091 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.091 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.091 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14695 2024-09-16T13:17:27.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.091 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:27.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636001+0000", 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0999999999999998e-07 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636004+0000", 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5959999999999998e-06 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636012+0000", 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636016+0000", 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9269999999999996e-06 2024-09-16T13:17:27.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648804+0000", 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012787543 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648873+0000", 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9263000000000004e-05 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648992+0000", 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119089 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14696 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2490368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36678209099999998, 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013132612, 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.095 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14696 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636023+0000", 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7300000000000005e-07 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636025+0000", 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.407e-06 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636033+0000", 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0129999999999993e-06 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636042+0000", 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1540000000000008e-06 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649027+0000", 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012985096 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649089+0000", 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1641000000000002e-05 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649155+0000", 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5528e-05 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14697 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:27.099 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.36675618999999998, 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013321177999999999, 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14697 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:27.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636049+0000", 2024-09-16T13:17:27.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:17:27.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636051+0000", 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.57e-06 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636059+0000", 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9440000000000005e-06 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636067+0000", 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0460000000000006e-06 2024-09-16T13:17:27.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649182+0000", 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01311522 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649245+0000", 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2930999999999998e-05 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649369+0000", 2024-09-16T13:17:27.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123681 2024-09-16T13:17:27.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.104 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.104 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.104 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:27.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:27.106 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:27.100+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:27.206 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:27.200+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:27.209 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:17:27.261 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:27.261 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:27.261 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:27.261 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:27.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:27.307 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:27.298+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:27.369 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone11 from 0xe6a4922 (overlap 0x165df64) to 0x0 2024-09-16T13:17:27.371 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:27.371 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:27.371 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:27.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:27.408 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:27.404+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:27.481 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:27.481 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:27.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:27.481 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:27.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.481 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1264 3.4 e27/23 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head v 27'4, mlcod=26'2)", 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.973739425000002, 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026410209999999999, 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:27.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501250+0000", 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2410000000000001e-06 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501259+0000", 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501277+0000", 2024-09-16T13:17:27.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8683999999999999e-05 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501288+0000", 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.065e-05 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501562+0000", 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027421200000000002 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501618+0000", 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6045000000000003e-05 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503841+0000", 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022230829999999998 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503890+0000", 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8526000000000003e-05 2024-09-16T13:17:27.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.467595737, 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:17:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.466810238000001, 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:17:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:17:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:17:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:17:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:17:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:17:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.465643316000001, 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:17:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.462768685, 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:17:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.459754945, 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:17:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:17:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:17:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.434550741999999, 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:17:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:17:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:17:27.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 22.668401852999999, 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:27.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:17:27.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:17:27.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:17:27.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5115 3.4 e34/23 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head v 34'14, mlcod=31'12)", 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.683353746999998, 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029400680000000001, 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791635+0000", 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4200000000000005e-07 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791643+0000", 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.157e-06 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791660+0000", 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6246e-05 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791666+0000", 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791743+0000", 2024-09-16T13:17:27.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7000000000000001e-05 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791756+0000", 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2729e-05 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794523+0000", 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027675949999999999 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794575+0000", 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1082000000000003e-05 2024-09-16T13:17:27.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5290 3.9 e34/23 3:975e9c33:::rbd_data.112d64676e7f.00000000000000b8:head v 34'26, mlcod=32'24)", 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.958105+0000", 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.516883328999999, 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002485054, 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958105+0000", 2024-09-16T13:17:27.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958105+0000", 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958106+0000", 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2139999999999999e-06 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958117+0000", 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1188e-05 2024-09-16T13:17:27.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958143+0000", 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5670999999999999e-05 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958152+0000", 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8389999999999996e-06 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958204+0000", 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2219999999999998e-05 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.958226+0000", 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2334e-05 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960538+0000", 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023112779999999999 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960590+0000", 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2309999999999999e-05 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 15.41629955, 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:17:27.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 15.41569011, 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:17:27.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:17:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:17:27.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6963 3.4 e37/23 3:2713f365:::rbd_data.112d6fb8d387.0000000000000133:head v 37'31, mlcod=37'29)", 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.919125960000001, 2024-09-16T13:17:27.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027598190000000002, 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555864+0000", 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555882+0000", 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7816e-05 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555909+0000", 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7838999999999999e-05 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555923+0000", 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3577000000000001e-05 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556083+0000", 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015987299999999999 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556132+0000", 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8655000000000001e-05 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558539+0000", 2024-09-16T13:17:27.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024076969999999999 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558622+0000", 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2777999999999999e-05 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:27.536 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.918406296000001, 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:17:27.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:17:27.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.91730548, 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.917165839999999, 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.548 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.916665309999999, 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:27.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:17:27.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:10843 3.0 e40/23 3:039a6044:::rbd_data.112ddef65b5a.0000000000000050:head v 40'68, mlcod=39'65)", 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.242323+0000", 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.232664892000001, 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025010879999999998, 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242323+0000", 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242323+0000", 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242324+0000", 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242332+0000", 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5529999999999993e-06 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242354+0000", 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1336999999999999e-05 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242362+0000", 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9859999999999994e-06 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242409+0000", 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7416000000000001e-05 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242431+0000", 2024-09-16T13:17:27.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2291000000000001e-05 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.244771+0000", 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023402280000000002 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.244824+0000", 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.9214959240000002, 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:17:27.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:14824 3.0 e49/23 3:0161fb21:::rbd_object_map.112d492326e3:head v 49'70, mlcod=40'68)", 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.316855+0000", 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.15813283, 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024824399999999998, 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.316855+0000", 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.316855+0000", 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.316856+0000", 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1039999999999999e-06 2024-09-16T13:17:27.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.316870+0000", 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3366999999999999e-05 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.316904+0000", 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4017999999999999e-05 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.316917+0000", 2024-09-16T13:17:27.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.358e-05 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.317092+0000", 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017441200000000001 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.317160+0000", 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8136e-05 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.319228+0000", 2024-09-16T13:17:27.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0020683759999999998 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.319338+0000", 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109447 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.566 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:27.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:27.569 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #10, image image_client.0-clone10 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent11 2024-09-16T13:17:27.569 INFO:teuthology.orchestra.run.smithi098.stdout:1886 read 0x70a4f6d thru 0x70b0cf9 (0xbd8d bytes) 2024-09-16T13:17:27.569 INFO:teuthology.orchestra.run.smithi098.stdout:1887 write 0x6489e6e thru 0x648c935 (0x2ac8 bytes) 2024-09-16T13:17:27.569 INFO:teuthology.orchestra.run.smithi098.stdout:1888 punch from 0x559b821 to 0x559e02f, (0x280e bytes) 2024-09-16T13:17:27.569 INFO:teuthology.orchestra.run.smithi098.stdout:1890 read 0x76ba2b3 thru 0x76bb9f5 (0x1743 bytes) 2024-09-16T13:17:27.570 INFO:teuthology.orchestra.run.smithi098.stdout:1891 read 0x4d10cbe thru 0x4d1abd7 (0x9f1a bytes) 2024-09-16T13:17:27.570 INFO:teuthology.orchestra.run.smithi098.stdout:1892 write 0x578faf7 thru 0x579cf95 (0xd49f bytes) 2024-09-16T13:17:27.570 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:27.570 INFO:teuthology.orchestra.run.smithi098.stdout:1896 read 0xe926de thru 0xe9ee16 (0xc739 bytes) 2024-09-16T13:17:27.570 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:27.570 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:27.559+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:27.583 INFO:teuthology.orchestra.run.smithi098.stdout:1898 trunc from 0xe6a4922 to 0x6a4fa58 2024-09-16T13:17:27.625 INFO:teuthology.orchestra.run.smithi098.stdout:1899 writesame 0xc853762 thru 0xc85aa53 data_size 0x1(0x72f2 bytes) 2024-09-16T13:17:27.625 INFO:teuthology.orchestra.run.smithi098.stdout:1900 read 0x500f6a9 thru 0x5016db1 (0x7709 bytes) 2024-09-16T13:17:27.639 INFO:teuthology.orchestra.run.smithi098.stdout:1901 trunc from 0xc85aa54 to 0x47398dd 2024-09-16T13:17:27.669 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:27.659+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:27.679 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:27.679 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:27.680 INFO:teuthology.orchestra.run.smithi098.stdout:1902 write 0xae78895 thru 0xae83c6d (0xb3d9 bytes) 2024-09-16T13:17:27.687 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:27.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:27.706 INFO:teuthology.orchestra.run.smithi098.stdout:1906 writesame 0xd2d3798 thru 0xd2dd3a8 data_size 0x1(0x9c11 bytes) 2024-09-16T13:17:27.706 INFO:teuthology.orchestra.run.smithi098.stdout:1907 write 0xb585e9b thru 0xb589f0d (0x4073 bytes) 2024-09-16T13:17:27.709 INFO:teuthology.orchestra.run.smithi098.stdout:1908 read 0x2e50e92 thru 0x2e5c01b (0xb18a bytes) 2024-09-16T13:17:27.709 INFO:teuthology.orchestra.run.smithi098.stdout:1910 write 0xb51ec67 thru 0xb51ff66 (0x1300 bytes) 2024-09-16T13:17:27.712 INFO:teuthology.orchestra.run.smithi098.stdout:1911 writesame 0x769ffe6 thru 0x76a314c data_size 0x1(0x3167 bytes) 2024-09-16T13:17:27.729 INFO:teuthology.orchestra.run.smithi098.stdout:1912 write 0x854ae47 thru 0x85576f9 (0xc8b3 bytes) 2024-09-16T13:17:27.733 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:27.744 INFO:teuthology.orchestra.run.smithi098.stdout:1915 trunc from 0xd2dd3a9 to 0x7a19e00 2024-09-16T13:17:27.770 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:27.763+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:27.778 INFO:teuthology.orchestra.run.smithi098.stdout:1916 trunc from 0x7a19e00 to 0xed5eefe 2024-09-16T13:17:27.791 INFO:teuthology.orchestra.run.smithi098.stdout:1918 punch from 0xa2a9660 to 0xa2ab7b6, (0x2156 bytes) 2024-09-16T13:17:27.791 INFO:teuthology.orchestra.run.smithi098.stdout:1919 write 0xdcbd6a1 thru 0xdcc5a77 (0x83d7 bytes) 2024-09-16T13:17:27.799 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:27.799 INFO:teuthology.orchestra.run.smithi098.stdout:1921 read 0x7e66dbf thru 0x7e75a26 (0xec68 bytes) 2024-09-16T13:17:27.800 INFO:teuthology.orchestra.run.smithi098.stdout:1922 write 0x18534c4 thru 0x1858934 (0x5471 bytes) 2024-09-16T13:17:27.816 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:27.817 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:27.817 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:27.817 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:27.817 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:27.817 INFO:teuthology.orchestra.run.smithi098.stdout:1925 write 0xdf549c4 thru 0xdf5d41b (0x8a58 bytes) 2024-09-16T13:17:27.821 INFO:teuthology.orchestra.run.smithi098.stdout:1928 write 0xa1f386c thru 0xa201b2e (0xe2c3 bytes) 2024-09-16T13:17:27.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:27.828 INFO:teuthology.orchestra.run.smithi098.stdout:1932 write 0xe5db3e1 thru 0xe5dfdcf (0x49ef bytes) 2024-09-16T13:17:27.831 INFO:teuthology.orchestra.run.smithi098.stdout:1933 clone 13 order 18 su 16384 sc 14 2024-09-16T13:17:27.871 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:27.864+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:27.924 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:27.924 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:27.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:27.924 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:27.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5101 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.139882385, 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595248, 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5101 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777333+0000", 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1949999999999996e-06 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777342+0000", 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6950000000000006e-06 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777348+0000", 2024-09-16T13:17:27.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785241+0000", 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078929329999999995 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785391+0000", 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150113 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785969+0000", 2024-09-16T13:17:27.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057822899999999998 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789745+0000", 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037756529999999999 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793823+0000", 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040786299999999998 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793880+0000", 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7157999999999997e-05 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793924+0000", 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3407e-05 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5102 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:27.930 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.139835092999999, 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016663710000000002, 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5102 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777377+0000", 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4600000000000003e-07 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777381+0000", 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5290000000000002e-06 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777393+0000", 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2001000000000001e-05 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777401+0000", 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3869999999999997e-06 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786753+0000", 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093523649999999996 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786860+0000", 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106921 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787286+0000", 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042651800000000003 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793815+0000", 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065290579999999999 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793932+0000", 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116688 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794002+0000", 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0313000000000003e-05 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794040+0000", 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7083999999999999e-05 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5103 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.139774206999999, 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017505544000000001, 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5103 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777438+0000", 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.78e-07 2024-09-16T13:17:27.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777442+0000", 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5950000000000001e-06 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777453+0000", 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0372e-05 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777458+0000", 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6559999999999997e-06 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788034+0000", 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010575753 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788111+0000", 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7038000000000003e-05 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788463+0000", 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000352028 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794263+0000", 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005800342 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794837+0000", 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057334300000000003 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794897+0000", 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0690000000000003e-05 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794942+0000", 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4848999999999997e-05 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5104 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:27.941 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.139729799000001, 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017694142999999999, 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5104 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777482+0000", 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777487+0000", 2024-09-16T13:17:27.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6639999999999997e-06 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777496+0000", 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2890000000000003e-06 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777501+0000", 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2730000000000002e-06 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788951+0000", 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011449429000000001 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789007+0000", 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5643999999999998e-05 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789205+0000", 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019820800000000001 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794268+0000", 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050636850000000001 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795094+0000", 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082569700000000002 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795139+0000", 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5055000000000001e-05 2024-09-16T13:17:27.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795175+0000", 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6149000000000002e-05 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5105 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.139667458000002, 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017729482000000001, 2024-09-16T13:17:27.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5105 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:27.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777544+0000", 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1100000000000004e-07 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777549+0000", 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777561+0000", 2024-09-16T13:17:27.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1524e-05 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777566+0000", 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.558e-06 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789781+0000", 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012214981 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789870+0000", 2024-09-16T13:17:27.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9171000000000004e-05 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790313+0000", 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044255299999999999 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794273+0000", 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039596129999999998 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795213+0000", 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094086099999999998 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795239+0000", 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5324e-05 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795273+0000", 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4198e-05 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5106 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.139622314, 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018621867, 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5106 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.953 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777590+0000", 2024-09-16T13:17:27.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777594+0000", 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6750000000000001e-06 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777604+0000", 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0076e-05 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777610+0000", 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7529999999999998e-06 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790893+0000", 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013282845 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790999+0000", 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106289 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791493+0000", 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049357800000000005 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794662+0000", 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031691409999999999 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796146+0000", 2024-09-16T13:17:27.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00148375 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796176+0000", 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0349e-05 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796211+0000", 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4530000000000003e-05 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5107 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.139591053, 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018745067000000001, 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5107 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777621+0000", 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-07 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777624+0000", 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7950000000000001e-06 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777633+0000", 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5979999999999997e-06 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777638+0000", 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.989e-06 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792024+0000", 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014385692 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792104+0000", 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0567000000000003e-05 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792340+0000", 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023624200000000001 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794667+0000", 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023269990000000002 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796304+0000", 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001636981 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796331+0000", 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6492999999999999e-05 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796365+0000", 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4102e-05 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5108 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.139537224000001, 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019090617000000001, 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5108 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777677+0000", 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7879999999999998e-06 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777686+0000", 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7679999999999997e-06 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777689+0000", 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.123e-06 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792751+0000", 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01506158 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792855+0000", 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010456 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793305+0000", 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045006100000000003 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794671+0000", 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001366304 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796393+0000", 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001721272 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796733+0000", 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000340611 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796764+0000", 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.099e-05 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5208 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.037777772999998, 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014882177999999999, 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5208 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879434+0000", 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.322e-06 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879443+0000", 2024-09-16T13:17:27.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:17:27.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879459+0000", 2024-09-16T13:17:27.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6133999999999998e-05 2024-09-16T13:17:27.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879467+0000", 2024-09-16T13:17:27.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2400000000000007e-06 2024-09-16T13:17:27.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888339+0000", 2024-09-16T13:17:27.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088721619999999994 2024-09-16T13:17:27.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888461+0000", 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121341 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888819+0000", 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035782799999999998 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891271+0000", 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002452872 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894224+0000", 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029522910000000001 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894275+0000", 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1647999999999997e-05 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894315+0000", 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9993999999999997e-05 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5210 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.037222584999999, 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01466423, 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5210 2024-09-16T13:17:27.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.977 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:27.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879989+0000", 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8800000000000001e-07 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879995+0000", 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880017+0000", 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2101000000000001e-05 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880024+0000", 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9240000000000001e-06 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890250+0000", 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010226012 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890344+0000", 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.446e-05 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890770+0000", 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042575099999999998 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894344+0000", 2024-09-16T13:17:27.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003574163 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894549+0000", 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020477599999999999 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894606+0000", 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6891000000000001e-05 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894653+0000", 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6867999999999999e-05 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5211 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.037178522000001, 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014800604, 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.982 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5211 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880033+0000", 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.06e-07 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880039+0000", 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880052+0000", 2024-09-16T13:17:27.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-05 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880059+0000", 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3090000000000001e-06 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891316+0000", 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011257118999999999 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891374+0000", 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8020999999999999e-05 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891577+0000", 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020371699999999999 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894348+0000", 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027708300000000002 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894726+0000", 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037819399999999999 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894784+0000", 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7667999999999997e-05 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894833+0000", 2024-09-16T13:17:27.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8936e-05 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5212 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.036823148, 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014645704000000001, 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.988 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5212 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880389+0000", 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1680000000000001e-06 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880395+0000", 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880408+0000", 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.308e-05 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880414+0000", 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.105e-06 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892037+0000", 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011622402 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892116+0000", 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9939999999999997e-05 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892597+0000", 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048076500000000002 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894492+0000", 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001895246 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894921+0000", 2024-09-16T13:17:27.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000428114 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894985+0000", 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4777999999999995e-05 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895034+0000", 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8133999999999997e-05 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:27.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.994 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5213 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:27.994 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:27.994 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.036689998, 2024-09-16T13:17:27.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014632608, 2024-09-16T13:17:27.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:27.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:27.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:27.995 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:27.995 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:27.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5213 2024-09-16T13:17:27.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.995 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:27.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:27.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880522+0000", 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2600000000000001e-07 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880527+0000", 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880538+0000", 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917999999999999e-05 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880543+0000", 2024-09-16T13:17:27.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.361e-06 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893069+0000", 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012525497999999999 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893152+0000", 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2805000000000004e-05 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893635+0000", 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048315100000000002 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894852+0000", 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001216806 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895075+0000", 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022300299999999999 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:27.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895122+0000", 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7077999999999999e-05 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895154+0000", 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1919999999999999e-05 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.000 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5214 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.036535347000001, 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017785977000000001, 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5214 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:28.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:28.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880681+0000", 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880689+0000", 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3610000000000001e-06 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880694+0000", 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5809999999999998e-06 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895186+0000", 2024-09-16T13:17:28.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014492430000000001 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895311+0000", 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012518300000000001 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895760+0000", 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044864900000000002 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898337+0000", 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025765229999999998 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898354+0000", 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7538999999999999e-05 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898406+0000", 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2054e-05 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898462+0000", 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5404e-05 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5215 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.036270513000002, 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018191319000000001, 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:28.006 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5215 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:28.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880941+0000", 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880946+0000", 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3710000000000002e-06 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880955+0000", 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1230000000000006e-06 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880959+0000", 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896782+0000", 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015822716000000001 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896876+0000", 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3667999999999998e-05 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.897289+0000", 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041357399999999999 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898474+0000", 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011847209999999999 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899036+0000", 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056215699999999998 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899090+0000", 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3662999999999998e-05 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899132+0000", 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1814000000000003e-05 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:28.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5282 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.967598399, 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014924988, 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5282 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.012 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:28.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.53e-07 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949635+0000", 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2422999999999999e-05 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949641+0000", 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.896e-06 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949644+0000", 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5270000000000001e-06 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959656+0000", 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010012378000000001 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959747+0000", 2024-09-16T13:17:28.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1273000000000005e-05 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960045+0000", 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029733 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963196+0000", 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031512620000000002 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964453+0000", 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001257357 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964502+0000", 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9044e-05 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964538+0000", 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5244999999999999e-05 2024-09-16T13:17:28.017 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5283 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.967563245000001, 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015065762, 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.018 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5283 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949650+0000", 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.736e-06 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949653+0000", 2024-09-16T13:17:28.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2969999999999999e-06 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949655+0000", 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1890000000000001e-06 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960643+0000", 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010988015 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:28.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960732+0000", 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9097000000000004e-05 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961195+0000", 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000462292 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963208+0000", 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002013641 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964585+0000", 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013765769999999999 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964669+0000", 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3863000000000007e-05 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964713+0000", 2024-09-16T13:17:28.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4706999999999997e-05 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5284 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.967553376000001, 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015152802, 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:28.024 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5284 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:28.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5899999999999998e-07 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949659+0000", 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.601e-06 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949672+0000", 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2364999999999999e-05 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949674+0000", 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961869+0000", 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012194903999999999 2024-09-16T13:17:28.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961971+0000", 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102154 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.962502+0000", 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00053037500000000003 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964751+0000", 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022487800000000001 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964755+0000", 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964789+0000", 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4054000000000003e-05 2024-09-16T13:17:28.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964810+0000", 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0922000000000001e-05 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5285 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.967534457999999, 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015553022, 2024-09-16T13:17:28.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5285 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949677+0000", 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6899999999999999e-07 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949679+0000", 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0090000000000002e-06 2024-09-16T13:17:28.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949682+0000", 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2550000000000002e-06 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949685+0000", 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.621e-06 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963409+0000", 2024-09-16T13:17:28.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013724485999999999 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963497+0000", 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7560999999999997e-05 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963933+0000", 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043592700000000002 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964820+0000", 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088772299999999998 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965165+0000", 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000344426 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965201+0000", 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5967000000000003e-05 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965230+0000", 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11223 3.8 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 33927~20122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.456362128, 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019333487, 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.036 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11223 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:28.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460851+0000", 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5730000000000002e-06 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464824+0000", 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039725730000000001 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464861+0000", 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7233000000000001e-05 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464872+0000", 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209000000000001e-05 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464987+0000", 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011423 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465188+0000", 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020091600000000001 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465929+0000", 2024-09-16T13:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00074102 2024-09-16T13:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.475697+0000", 2024-09-16T13:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009768367 2024-09-16T13:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480068+0000", 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043713290000000002 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480143+0000", 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5000999999999996e-05 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480182+0000", 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9035999999999997e-05 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:28.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:28.042 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:28.042 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:28.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:28.044 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:28.035+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:28.144 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:28.140+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:28.149 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:28.149 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:28.149 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:28.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:28.245 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:28.235+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:28.265 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:28.265 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:28.265 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:28.265 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:28.265 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:28.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:28.346 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:28.340+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:28.382 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8369 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.318580582999999, 2024-09-16T13:17:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020058919000000001, 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8369 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056335+0000", 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056338+0000", 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056344+0000", 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8150000000000002e-06 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056348+0000", 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.070951+0000", 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014603055 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071060+0000", 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109462 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071508+0000", 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044824200000000001 2024-09-16T13:17:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074304+0000", 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027951719999999998 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076331+0000", 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020276529999999999 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076369+0000", 2024-09-16T13:17:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7557e-05 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076393+0000", 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4445999999999999e-05 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8370 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.318563265, 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021240851000000002, 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8370 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056355+0000", 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.667e-06 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056360+0000", 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6620000000000002e-06 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056366+0000", 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072672+0000", 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016306157000000002 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072746+0000", 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3648999999999999e-05 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.073364+0000", 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061885499999999997 2024-09-16T13:17:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076445+0000", 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003080396 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077482+0000", 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001036709 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077535+0000", 2024-09-16T13:17:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3211000000000003e-05 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077592+0000", 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7664000000000002e-05 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8371 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.318534469999999, 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021386721000000001, 2024-09-16T13:17:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8371 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056381+0000", 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056385+0000", 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056391+0000", 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6959999999999999e-06 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056396+0000", 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2549999999999999e-06 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074355+0000", 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017959098999999999 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074503+0000", 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014797200000000001 2024-09-16T13:17:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074955+0000", 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045213300000000002 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077607+0000", 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026519500000000001 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077660+0000", 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077725+0000", 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4991000000000002e-05 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077767+0000", 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8372 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.318513448999999, 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021557531000000001, 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056402+0000", 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1399999999999997e-07 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056405+0000", 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6520000000000002e-06 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056411+0000", 2024-09-16T13:17:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2009999999999996e-06 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056415+0000", 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.211e-06 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075482+0000", 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019067163000000002 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075546+0000", 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3516000000000006e-05 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075844+0000", 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029810999999999998 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077784+0000", 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019399350000000001 2024-09-16T13:17:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077823+0000", 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9524000000000003e-05 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077917+0000", 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4090000000000002e-05 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077959+0000", 2024-09-16T13:17:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1615000000000001e-05 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8373 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.318493106, 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022897421000000001, 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:28.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3200000000000005e-07 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056427+0000", 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-06 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056433+0000", 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056437+0000", 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076470+0000", 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020032239 2024-09-16T13:17:28.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076524+0000", 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4475000000000003e-05 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076996+0000", 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000471905 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.078749+0000", 2024-09-16T13:17:28.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001752539 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079217+0000", 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046819400000000001 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079276+0000", 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9407000000000002e-05 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079319+0000", 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8398 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.289129314, 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017439282, 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8398 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085788+0000", 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4719999999999998e-06 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085793+0000", 2024-09-16T13:17:28.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6179999999999997e-06 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085796+0000", 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6680000000000001e-06 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099514+0000", 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013717895000000001 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099873+0000", 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035970799999999998 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100269+0000", 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000395382 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103068+0000", 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027992680000000002 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103114+0000", 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6189000000000001e-05 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103178+0000", 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3845999999999998e-05 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103225+0000", 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6888999999999998e-05 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.418 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8399 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.289087837, 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017551573000000001, 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8399 2024-09-16T13:17:28.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085828+0000", 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085830+0000", 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8609999999999998e-06 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085835+0000", 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.437e-06 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085837+0000", 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.571e-06 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100950+0000", 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015112537000000001 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101047+0000", 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6945000000000001e-05 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101468+0000", 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000420794 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103315+0000", 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018474769999999999 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103321+0000", 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103356+0000", 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4764000000000001e-05 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103379+0000", 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3102999999999999e-05 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.424 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8400 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.289074936, 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017960752, 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8400 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.53e-07 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085842+0000", 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2620000000000001e-06 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085847+0000", 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2119999999999997e-06 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085849+0000", 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.204e-06 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101995+0000", 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016146384999999999 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102100+0000", 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010482199999999999 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102466+0000", 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000365731 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103389+0000", 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092367199999999995 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103702+0000", 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031220400000000001 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103754+0000", 2024-09-16T13:17:28.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2318999999999998e-05 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103801+0000", 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6687999999999998e-05 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8539 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:28.430 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.130943381, 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018486425000000001, 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8539 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0500000000000003e-07 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243975+0000", 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800000000000002e-06 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243982+0000", 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243986+0000", 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4100000000000001e-06 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258523+0000", 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014536604999999999 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258678+0000", 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155341 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.259218+0000", 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053930600000000003 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262325+0000", 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031076799999999998 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262367+0000", 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1480000000000003e-05 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262417+0000", 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0555e-05 2024-09-16T13:17:28.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262458+0000", 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0556999999999998e-05 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8540 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.130923960000001, 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018906683000000001, 2024-09-16T13:17:28.436 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8540 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243992+0000", 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.75e-07 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243995+0000", 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1219999999999999e-06 2024-09-16T13:17:28.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244001+0000", 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5010000000000001e-06 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244006+0000", 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999996e-06 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260137+0000", 2024-09-16T13:17:28.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016131 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260200+0000", 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3782999999999996e-05 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260598+0000", 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039746600000000002 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262331+0000", 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017327849999999999 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262786+0000", 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045546300000000003 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262851+0000", 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-05 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262898+0000", 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7098000000000002e-05 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8541 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.130904737, 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019366889000000002, 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8541 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:28.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244011+0000", 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244014+0000", 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.79e-06 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244020+0000", 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6370000000000002e-06 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244025+0000", 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3359999999999997e-06 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261122+0000", 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017097161 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261228+0000", 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105989 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261475+0000", 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024757000000000002 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263022+0000", 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015466760000000001 2024-09-16T13:17:28.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263300+0000", 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027828100000000001 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263344+0000", 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4008999999999998e-05 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263377+0000", 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2832e-05 2024-09-16T13:17:28.447 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11776 3.1a 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.857501729999999, 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027432719000000001, 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.448 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11776 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517415+0000", 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5239999999999999e-06 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526328+0000", 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089138059999999998 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526362+0000", 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3775000000000001e-05 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526379+0000", 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6835000000000001e-05 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526467+0000", 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7703999999999999e-05 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526653+0000", 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186379 2024-09-16T13:17:28.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.527624+0000", 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097080400000000002 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543313+0000", 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015689456000000001 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544696+0000", 2024-09-16T13:17:28.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013822979999999999 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544784+0000", 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8419999999999997e-05 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544846+0000", 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1717999999999994e-05 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:11778 3.1 e40/23 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head v 40'57, mlcod=40'55)", 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.851017703, 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01745787, 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:28.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523898+0000", 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-06 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533067+0000", 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091691120000000001 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533101+0000", 2024-09-16T13:17:28.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3733999999999999e-05 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533115+0000", 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3410999999999999e-05 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533194+0000", 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9827000000000007e-05 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533223+0000", 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8937000000000001e-05 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541297+0000", 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080739850000000005 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541355+0000", 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7708999999999999e-05 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.457 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11777 3.1a 3:59cb7a3f:::rbd_object_map.112d21cb8818:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.848512791999999, 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018649718999999999, 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11777 2024-09-16T13:17:28.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526404+0000", 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.654e-06 2024-09-16T13:17:28.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526409+0000", 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.882e-06 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526424+0000", 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5328999999999998e-05 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526428+0000", 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6219999999999998e-06 2024-09-16T13:17:28.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542209+0000", 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015781765999999999 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542334+0000", 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125148 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542804+0000", 2024-09-16T13:17:28.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046944899999999998 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543732+0000", 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092824699999999999 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544929+0000", 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001196522 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545004+0000", 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4962999999999994e-05 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545052+0000", 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8136999999999997e-05 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11779 3.9 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.82799121, 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020806181999999999, 2024-09-16T13:17:28.463 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11779 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546926+0000", 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9769999999999999e-06 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552330+0000", 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054043980000000004 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552353+0000", 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2875999999999999e-05 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552362+0000", 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7299999999999994e-06 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552447+0000", 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5620999999999999e-05 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552710+0000", 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026273300000000002 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.553446+0000", 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073563299999999997 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.559002+0000", 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055564580000000002 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567615+0000", 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086126680000000004 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567689+0000", 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3839999999999998e-05 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567730+0000", 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1248000000000002e-05 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13736 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.0769200330000004, 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018229967999999999, 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13736 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2e-07 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297998+0000", 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.824e-06 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298004+0000", 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8730000000000004e-06 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298008+0000", 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312651+0000", 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014643351000000001 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312740+0000", 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8466000000000003e-05 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:28.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313081+0000", 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034067899999999998 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315982+0000", 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002900943 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316051+0000", 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9029999999999995e-05 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316101+0000", 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0680000000000003e-05 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316225+0000", 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012359099999999999 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13737 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.0769029129999996, 2024-09-16T13:17:28.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018381208999999999, 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13737 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298015+0000", 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6869999999999999e-06 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298021+0000", 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7799999999999997e-06 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298027+0000", 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.071e-06 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313829+0000", 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015801696 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314019+0000", 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018995700000000001 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314379+0000", 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036036500000000002 2024-09-16T13:17:28.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315988+0000", 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016084949999999999 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316290+0000", 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000302732 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316324+0000", 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3937999999999999e-05 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316393+0000", 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8966000000000003e-05 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13738 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.0768839310000002, 2024-09-16T13:17:28.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021400311000000002, 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13738 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298032+0000", 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7400000000000003e-07 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298034+0000", 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5770000000000001e-06 2024-09-16T13:17:28.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298040+0000", 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7010000000000001e-06 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298044+0000", 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2069999999999997e-06 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315033+0000", 2024-09-16T13:17:28.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016988970999999999 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315106+0000", 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2723000000000002e-05 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315428+0000", 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032191099999999997 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319249+0000", 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003821166 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319313+0000", 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319347+0000", 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4591999999999997e-05 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319431+0000", 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3893000000000005e-05 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13739 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.0768664189999999, 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021506517999999999, 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.486 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13739 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298049+0000", 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298052+0000", 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.819e-06 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298058+0000", 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6960000000000002e-06 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298061+0000", 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7560000000000001e-06 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316465+0000", 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018403744 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316747+0000", 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028175200000000002 2024-09-16T13:17:28.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.317694+0000", 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000947115 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319304+0000", 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016102029999999999 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319446+0000", 2024-09-16T13:17:28.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142008 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319490+0000", 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3690999999999999e-05 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319555+0000", 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5191000000000007e-05 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13740 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.0768129359999996, 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023881870999999999, 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13740 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:28.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298103+0000", 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:28.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298106+0000", 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1559999999999999e-06 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298120+0000", 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4474999999999999e-05 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298124+0000", 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1699999999999999e-06 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:28.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319635+0000", 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021510129999999999 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319762+0000", 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012746400000000001 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.320225+0000", 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046264800000000002 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:28.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321629+0000", 2024-09-16T13:17:28.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00140383 2024-09-16T13:17:28.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321682+0000", 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3720000000000001e-05 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321757+0000", 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321984+0000", 2024-09-16T13:17:28.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022684199999999999 2024-09-16T13:17:28.498 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.498 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.498 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.498 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:28.498 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:28.498 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:28.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:28.499 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone12 from 0xed5eefe (overlap 0x47398dd) to 0x239cc6e 2024-09-16T13:17:28.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:28.501 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:28.491+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:28.601 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:28.596+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:28.606 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #11, image image_client.0-clone11 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent12 2024-09-16T13:17:28.609 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:28.610 INFO:teuthology.orchestra.run.smithi098.stdout:1935 write 0x171e3b8 thru 0x1728b6b (0xa7b4 bytes) 2024-09-16T13:17:28.619 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:28.620 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:28.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:28.639 INFO:teuthology.orchestra.run.smithi098.stdout:1936 write 0xc24ac6c thru 0xc24fd4c (0x50e1 bytes) 2024-09-16T13:17:28.646 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:28.647 INFO:teuthology.orchestra.run.smithi098.stdout:1939 punch from 0xb60a4e6 to 0xb6157e6, (0xb300 bytes) 2024-09-16T13:17:28.647 INFO:teuthology.orchestra.run.smithi098.stdout:1940 write 0x1fd5b2e thru 0x1fd74d0 (0x19a3 bytes) 2024-09-16T13:17:28.647 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:28.657 INFO:teuthology.orchestra.run.smithi098.stdout:1942 read 0xe6b1ed9 thru 0xe6b9b68 (0x7c90 bytes) 2024-09-16T13:17:28.658 INFO:teuthology.orchestra.run.smithi098.stdout:1943 writesame 0xa868ba4 thru 0xa8744d8 data_size 0x1(0xb935 bytes) 2024-09-16T13:17:28.673 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:28.693 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:28.699 INFO:teuthology.orchestra.run.smithi098.stdout:1944 write 0x5f82dfa thru 0x5f8ddcf (0xafd6 bytes) 2024-09-16T13:17:28.701 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:28.696+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:28.713 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:28.719 INFO:teuthology.orchestra.run.smithi098.stdout:1945 writesame 0x2eff3d2 thru 0x2effae2 data_size 0x9(0x711 bytes) 2024-09-16T13:17:28.726 INFO:teuthology.orchestra.run.smithi098.stdout:1946 write 0xec85995 thru 0xec8dc86 (0x82f2 bytes) 2024-09-16T13:17:28.734 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:28.734 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:28.735 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:28.735 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:28.735 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:28.735 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:28.735 INFO:teuthology.orchestra.run.smithi098.stdout:1951 read 0x2bd273 thru 0x2c95b0 (0xc33e bytes) 2024-09-16T13:17:28.737 INFO:teuthology.orchestra.run.smithi098.stdout:1952 write 0x92bfc16 thru 0x92c8a07 (0x8df2 bytes) 2024-09-16T13:17:28.739 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:28.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:28.759 INFO:teuthology.orchestra.run.smithi098.stdout:1955 read 0x23b4479 thru 0x23b6f0f (0x2a97 bytes) 2024-09-16T13:17:28.760 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:28.760 INFO:teuthology.orchestra.run.smithi098.stdout:1956 write 0x69dced6 thru 0x69eb160 (0xe28b bytes) 2024-09-16T13:17:28.779 INFO:teuthology.orchestra.run.smithi098.stdout:1957 write 0x58857ad thru 0x5893eda (0xe72e bytes) 2024-09-16T13:17:28.784 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:28.796 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:28.796 INFO:teuthology.orchestra.run.smithi098.stdout:1961 write 0xc4b30b5 thru 0xc4b801f (0x4f6b bytes) 2024-09-16T13:17:28.803 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:28.796+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:28.806 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:28.808 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:28.809 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:28.813 INFO:teuthology.orchestra.run.smithi098.stdout:1962 write 0xf4d839 thru 0xf5900e (0xb7d6 bytes) 2024-09-16T13:17:28.826 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:28.826 INFO:teuthology.orchestra.run.smithi098.stdout:1967 writesame 0x46aeb74 thru 0x46b81e3 data_size 0x74(0x9670 bytes) 2024-09-16T13:17:28.843 INFO:teuthology.orchestra.run.smithi098.stdout:1968 writesame 0x9dbe1af thru 0x9dcd34c data_size 0xa9(0xf19e bytes) 2024-09-16T13:17:28.855 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:28.855 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:28.857 INFO:teuthology.orchestra.run.smithi098.stdout:1969 write 0xd0c44c1 thru 0xd0d3396 (0xeed6 bytes) 2024-09-16T13:17:28.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:28.869 INFO:teuthology.orchestra.run.smithi098.stdout:1970 write 0x88bcbb9 thru 0x88c11f1 (0x4639 bytes) 2024-09-16T13:17:28.893 INFO:teuthology.orchestra.run.smithi098.stdout:1971 trunc from 0xed5eefe to 0xa07d24d 2024-09-16T13:17:28.904 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:28.895+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:28.940 INFO:teuthology.orchestra.run.smithi098.stdout:1972 read 0xb59cb9 thru 0xb5cbcc (0x2f14 bytes) 2024-09-16T13:17:28.941 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:28.941 INFO:teuthology.orchestra.run.smithi098.stdout:1976 write 0x6fbcf5b thru 0x6fc5e22 (0x8ec8 bytes) 2024-09-16T13:17:28.965 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:28.965 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:28.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:28.977 INFO:teuthology.orchestra.run.smithi098.stdout:1977 write 0xe8f9db9 thru 0xe9019d5 (0x7c1d bytes) 2024-09-16T13:17:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:28.988 INFO:teuthology.orchestra.run.smithi098.stdout:1980 writesame 0x35df9ee thru 0x35ebf1c data_size 0xd(0xc52f bytes) 2024-09-16T13:17:29.005 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:29.000+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:29.008 INFO:teuthology.orchestra.run.smithi098.stdout:1981 punch from 0x69c4b50 to 0x69d0a21, (0xbed1 bytes) 2024-09-16T13:17:29.008 INFO:teuthology.orchestra.run.smithi098.stdout:1982 write 0xb66e5c1 thru 0xb67d341 (0xed81 bytes) 2024-09-16T13:17:29.045 INFO:teuthology.orchestra.run.smithi098.stdout:1983 trunc from 0xe9019d6 to 0xb2cea8 2024-09-16T13:17:29.105 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:29.099+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:29.112 INFO:teuthology.orchestra.run.smithi098.stdout:1984 trunc from 0xb2cea8 to 0x1adac8 2024-09-16T13:17:29.130 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:29.130 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:29.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:29.206 INFO:teuthology.orchestra.run.smithi098.stdout:1985 trunc from 0x1adac8 to 0x2b90c0c 2024-09-16T13:17:29.206 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:29.200+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:29.216 INFO:teuthology.orchestra.run.smithi098.stdout:1986 read 0x1be4298 thru 0x1bed43c (0x91a5 bytes) 2024-09-16T13:17:29.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:17:29.227 INFO:teuthology.orchestra.run.smithi098.stdout:1987 trunc from 0x2b90c0c to 0x7032fcb 2024-09-16T13:17:29.241 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:29.241 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:29.248 INFO:teuthology.orchestra.run.smithi098.stdout:1988 trunc from 0x7032fcb to 0xc0b5498 2024-09-16T13:17:29.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:29.277 INFO:teuthology.orchestra.run.smithi098.stdout:1989 trunc from 0xc0b5498 to 0x157cedf 2024-09-16T13:17:29.287 INFO:teuthology.orchestra.run.smithi098.stdout:1990 read 0x1047ab3 thru 0x104850e (0xa5c bytes) 2024-09-16T13:17:29.287 INFO:teuthology.orchestra.run.smithi098.stdout:1991 read 0x66fe20 thru 0x67deda (0xe0bb bytes) 2024-09-16T13:17:29.291 INFO:teuthology.orchestra.run.smithi098.stdout:1993 trunc from 0x157cedf to 0x3ed8532 2024-09-16T13:17:29.307 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:29.300+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:29.318 INFO:teuthology.orchestra.run.smithi098.stdout:1994 write 0x61aff02 thru 0x61b2dd7 (0x2ed6 bytes) 2024-09-16T13:17:29.326 INFO:teuthology.orchestra.run.smithi098.stdout:1996 read 0x2ef10ab thru 0x2efc4f4 (0xb44a bytes) 2024-09-16T13:17:29.326 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:29.326 INFO:teuthology.orchestra.run.smithi098.stdout:1998 read 0x15aaf5e thru 0x15b3947 (0x89ea bytes) 2024-09-16T13:17:29.327 INFO:teuthology.orchestra.run.smithi098.stdout:1999 write 0x38d8dcb thru 0x38dcce3 (0x3f19 bytes) 2024-09-16T13:17:29.333 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:29.333 INFO:teuthology.orchestra.run.smithi098.stdout:2001 punch from 0x1ebac6d to 0x1ec717c, (0xc50f bytes) 2024-09-16T13:17:29.333 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:29.349 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:29.349 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:29.356 INFO:teuthology.orchestra.run.smithi098.stdout:2003 writesame 0xdf5df0d thru 0xdf65061 data_size 0x13(0x7155 bytes) 2024-09-16T13:17:29.356 INFO:teuthology.orchestra.run.smithi098.stdout:2004 write 0x572ec7c thru 0x5739b8e (0xaf13 bytes) 2024-09-16T13:17:29.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:29.366 INFO:teuthology.orchestra.run.smithi098.stdout:2005 write 0x28a59f thru 0x293ba0 (0x9602 bytes) 2024-09-16T13:17:29.369 INFO:teuthology.orchestra.run.smithi098.stdout:2006 writesame 0xa6f6b8e thru 0xa700270 data_size 0x13(0x96e3 bytes) 2024-09-16T13:17:29.380 INFO:teuthology.orchestra.run.smithi098.stdout:2008 writesame 0xc96bf21 thru 0xc974880 data_size 0x70(0x8960 bytes) 2024-09-16T13:17:29.391 INFO:teuthology.orchestra.run.smithi098.stdout:2009 read 0x4c4b339 thru 0x4c52954 (0x761c bytes) 2024-09-16T13:17:29.408 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:29.404+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:29.411 INFO:teuthology.orchestra.run.smithi098.stdout:2010 trunc from 0xdf65062 to 0x20a7141 2024-09-16T13:17:29.453 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:29.453 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:29.458 INFO:teuthology.orchestra.run.smithi098.stdout:2011 writesame 0xceff6b1 thru 0xcf03b5c data_size 0x1e(0x44ac bytes) 2024-09-16T13:17:29.458 INFO:teuthology.orchestra.run.smithi098.stdout:2012 read 0xa532bf6 thru 0xa536e96 (0x42a1 bytes) 2024-09-16T13:17:29.458 INFO:teuthology.orchestra.run.smithi098.stdout:2013 read 0x337ce1a thru 0x3389d6e (0xcf55 bytes) 2024-09-16T13:17:29.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:29.472 INFO:teuthology.orchestra.run.smithi098.stdout:2014 trunc from 0xcf03b5d to 0x3bb3cb9 2024-09-16T13:17:29.482 INFO:teuthology.orchestra.run.smithi098.stdout:2017 punch from 0x2b6d76d to 0x2b7a432, (0xccc5 bytes) 2024-09-16T13:17:29.482 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:29.504 INFO:teuthology.orchestra.run.smithi098.stdout:2019 writesame 0xc347dbe thru 0xc35293d data_size 0x20(0xab80 bytes) 2024-09-16T13:17:29.509 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:29.504+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:29.517 INFO:teuthology.orchestra.run.smithi098.stdout:2020 write 0xdd94b11 thru 0xdd97a81 (0x2f71 bytes) 2024-09-16T13:17:29.523 INFO:teuthology.orchestra.run.smithi098.stdout:2021 read 0xcc292e9 thru 0xcc31ab6 (0x87ce bytes) 2024-09-16T13:17:29.524 INFO:teuthology.orchestra.run.smithi098.stdout:2022 read 0x428cd83 thru 0x42976cb (0xa949 bytes) 2024-09-16T13:17:29.530 INFO:teuthology.orchestra.run.smithi098.stdout:2023 trunc from 0xdd97a82 to 0xa170f71 2024-09-16T13:17:29.550 INFO:teuthology.orchestra.run.smithi098.stdout:2025 punch from 0x6f8c44a to 0x6f8f6b1, (0x3267 bytes) 2024-09-16T13:17:29.558 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:29.558 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:29.562 INFO:teuthology.orchestra.run.smithi098.stdout:2026 write 0xb33353f thru 0xb34082b (0xd2ed bytes) 2024-09-16T13:17:29.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:29.575 INFO:teuthology.orchestra.run.smithi098.stdout:2027 punch from 0x9ab68ab to 0x9ac4f9c, (0xe6f1 bytes) 2024-09-16T13:17:29.575 INFO:teuthology.orchestra.run.smithi098.stdout:2028 punch from 0xab1265a to 0xab15cc5, (0x366b bytes) 2024-09-16T13:17:29.583 INFO:teuthology.orchestra.run.smithi098.stdout:2029 writesame 0xb682a00 thru 0xb68fc45 data_size 0x23(0xd246 bytes) 2024-09-16T13:17:29.583 INFO:teuthology.orchestra.run.smithi098.stdout:2030 writesame 0x43211b0 thru 0x4327882 data_size 0x1(0x66d3 bytes) 2024-09-16T13:17:29.606 INFO:teuthology.orchestra.run.smithi098.stdout:2031 read 0xa953f18 thru 0xa95c021 (0x810a bytes) 2024-09-16T13:17:29.610 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:29.604+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:29.622 INFO:teuthology.orchestra.run.smithi098.stdout:2032 trunc from 0xb68fc46 to 0x1c3614e 2024-09-16T13:17:29.649 INFO:teuthology.orchestra.run.smithi098.stdout:2033 trunc from 0x1c3614e to 0x480df8b 2024-09-16T13:17:29.661 INFO:teuthology.orchestra.run.smithi098.stdout:2034 trunc from 0x480df8b to 0x5381deb 2024-09-16T13:17:29.669 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:29.669 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:29.672 INFO:teuthology.orchestra.run.smithi098.stdout:2036 read 0x5313741 thru 0x531a2f3 (0x6bb3 bytes) 2024-09-16T13:17:29.672 INFO:teuthology.orchestra.run.smithi098.stdout:2039 read 0x4267be1 thru 0x426ea72 (0x6e92 bytes) 2024-09-16T13:17:29.672 INFO:teuthology.orchestra.run.smithi098.stdout:2041 read 0x1180935 thru 0x1187ab1 (0x717d bytes) 2024-09-16T13:17:29.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:29.700 INFO:teuthology.orchestra.run.smithi098.stdout:2043 writesame 0xe28f156 thru 0xe298452 data_size 0x6f(0x92fd bytes) 2024-09-16T13:17:29.711 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:29.703+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:29.722 INFO:teuthology.orchestra.run.smithi098.stdout:2045 trunc from 0xe298453 to 0x14e1498 2024-09-16T13:17:29.731 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:29.732 INFO:teuthology.orchestra.run.smithi098.stdout:2047 trunc from 0x14e1498 to 0xbd2d17 2024-09-16T13:17:29.744 INFO:teuthology.orchestra.run.smithi098.stdout:2049 read 0x654418 thru 0x657bf1 (0x37da bytes) 2024-09-16T13:17:29.763 INFO:teuthology.orchestra.run.smithi098.stdout:2050 writesame 0x5c7c144 thru 0x5c7d3c6 data_size 0x1(0x1283 bytes) 2024-09-16T13:17:29.764 INFO:teuthology.orchestra.run.smithi098.stdout:2051 trunc from 0x5c7d3c7 to 0x530224c 2024-09-16T13:17:29.780 INFO:teuthology.orchestra.run.smithi098.stdout:2052 trunc from 0x530224c to 0x210aba4 2024-09-16T13:17:29.793 INFO:teuthology.orchestra.run.smithi098.stdout:2053 read 0x1915c8f thru 0x19237f7 (0xdb69 bytes) 2024-09-16T13:17:29.806 INFO:teuthology.orchestra.run.smithi098.stdout:2057 write 0x2578f0f thru 0x2587166 (0xe258 bytes) 2024-09-16T13:17:29.812 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:29.803+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:29.820 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:29.820 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:29.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:29.837 INFO:teuthology.orchestra.run.smithi098.stdout:2058 write 0x8e31c24 thru 0x8e3dd15 (0xc0f2 bytes) 2024-09-16T13:17:29.849 INFO:teuthology.orchestra.run.smithi098.stdout:2059 write 0x5d1e0e3 thru 0x5d26486 (0x83a4 bytes) 2024-09-16T13:17:29.861 INFO:teuthology.orchestra.run.smithi098.stdout:2060 writesame 0x11c7cbb thru 0x11d170d data_size 0xd(0x9a53 bytes) 2024-09-16T13:17:29.876 INFO:teuthology.orchestra.run.smithi098.stdout:2061 writesame 0x7c315c8 thru 0x7c3ab1d data_size 0x5(0x9556 bytes) 2024-09-16T13:17:29.899 INFO:teuthology.orchestra.run.smithi098.stdout:2062 trunc from 0x8e3dd16 to 0xe64a40a 2024-09-16T13:17:29.908 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:29.908 INFO:teuthology.orchestra.run.smithi098.stdout:2066 writesame 0x9658c2d thru 0x9658ca5 data_size 0x1(0x79 bytes) 2024-09-16T13:17:29.913 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:29.903+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:29.914 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:29.924 INFO:teuthology.orchestra.run.smithi098.stdout:2069 trunc from 0xe64a40a to 0x82829d3 2024-09-16T13:17:29.946 INFO:teuthology.orchestra.run.smithi098.stdout:2071 read 0x5ffe39f thru 0x60008b4 (0x2516 bytes) 2024-09-16T13:17:29.946 INFO:teuthology.orchestra.run.smithi098.stdout:2073 read 0x42df3ca thru 0x42ed168 (0xdd9f bytes) 2024-09-16T13:17:29.946 INFO:teuthology.orchestra.run.smithi098.stdout:2076 read 0x37d4f97 thru 0x37d8755 (0x37bf bytes) 2024-09-16T13:17:29.948 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:29.949 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:29.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:29.967 INFO:teuthology.orchestra.run.smithi098.stdout:2078 write 0xea7e825 thru 0xea807fa (0x1fd6 bytes) 2024-09-16T13:17:29.976 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:29.977 INFO:teuthology.orchestra.run.smithi098.stdout:2083 writesame 0x677d8f6 thru 0x6782222 data_size 0xd(0x492d bytes) 2024-09-16T13:17:29.984 INFO:teuthology.orchestra.run.smithi098.stdout:2084 writesame 0xa898caa thru 0xa89f92b data_size 0x2b(0x6c82 bytes) 2024-09-16T13:17:29.993 INFO:teuthology.orchestra.run.smithi098.stdout:2085 punch from 0x7f129ea to 0x7f20559, (0xdb6f bytes) 2024-09-16T13:17:29.993 INFO:teuthology.orchestra.run.smithi098.stdout:2086 punch from 0xdcfea87 to 0xdd08391, (0x990a bytes) 2024-09-16T13:17:29.993 INFO:teuthology.orchestra.run.smithi098.stdout:2087 writesame 0x11098e3 thru 0x110c83d data_size 0x9(0x2f5b bytes) 2024-09-16T13:17:30.002 INFO:teuthology.orchestra.run.smithi098.stdout:2089 read 0x85b1a0f thru 0x85c04b2 (0xeaa4 bytes) 2024-09-16T13:17:30.002 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.002 INFO:teuthology.orchestra.run.smithi098.stdout:2091 punch from 0x709d620 to 0x70ad0e6, (0xfac6 bytes) 2024-09-16T13:17:30.002 INFO:teuthology.orchestra.run.smithi098.stdout:2092 read 0x5a79d16 thru 0x5a7f252 (0x553d bytes) 2024-09-16T13:17:30.002 INFO:teuthology.orchestra.run.smithi098.stdout:2094 punch from 0x7d50fad to 0x7d564b6, (0x5509 bytes) 2024-09-16T13:17:30.003 INFO:teuthology.orchestra.run.smithi098.stdout:2095 write 0x5667048 thru 0x5673aa1 (0xca5a bytes) 2024-09-16T13:17:30.014 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:30.007+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:30.017 INFO:teuthology.orchestra.run.smithi098.stdout:2096 write 0xe7a6279 thru 0xe7b5436 (0xf1be bytes) 2024-09-16T13:17:30.029 INFO:teuthology.orchestra.run.smithi098.stdout:2097 write 0x72e4ce3 thru 0x72ee32e (0x964c bytes) 2024-09-16T13:17:30.037 INFO:teuthology.orchestra.run.smithi098.stdout:2098 read 0x5ed16bd thru 0x5ed8723 (0x7067 bytes) 2024-09-16T13:17:30.038 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.038 INFO:teuthology.orchestra.run.smithi098.stdout:2100 write 0x7a608fc thru 0x7a634c6 (0x2bcb bytes) 2024-09-16T13:17:30.042 INFO:teuthology.orchestra.run.smithi098.stdout:2101 write 0xcbbf6ff thru 0xcbccbb7 (0xd4b9 bytes) 2024-09-16T13:17:30.053 INFO:teuthology.orchestra.run.smithi098.stdout:2102 writesame 0x6e1034d thru 0x6e1aab7 data_size 0x1(0xa76b bytes) 2024-09-16T13:17:30.073 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:30.073 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:30.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:30.085 INFO:teuthology.orchestra.run.smithi098.stdout:2104 writesame 0xadfb87a thru 0xae07495 data_size 0x6(0xbc1c bytes) 2024-09-16T13:17:30.097 INFO:teuthology.orchestra.run.smithi098.stdout:2105 read 0x187304f thru 0x1876d27 (0x3cd9 bytes) 2024-09-16T13:17:30.097 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.097 INFO:teuthology.orchestra.run.smithi098.stdout:2107 read 0xd2055c9 thru 0xd213493 (0xdecb bytes) 2024-09-16T13:17:30.097 INFO:teuthology.orchestra.run.smithi098.stdout:2108 write 0x2e8f0f4 thru 0x2e90143 (0x1050 bytes) 2024-09-16T13:17:30.105 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.105 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.105 INFO:teuthology.orchestra.run.smithi098.stdout:2112 read 0x5c8049b thru 0x5c89577 (0x90dd bytes) 2024-09-16T13:17:30.105 INFO:teuthology.orchestra.run.smithi098.stdout:2113 write 0x7f8c1fa thru 0x7f96068 (0x9e6f bytes) 2024-09-16T13:17:30.114 INFO:teuthology.orchestra.run.smithi098.stdout:2117 read 0x588d370 thru 0x5897f7c (0xac0d bytes) 2024-09-16T13:17:30.114 INFO:teuthology.orchestra.run.smithi098.stdout:2118 punch from 0xc85dc3e to 0xc8630fb, (0x54bd bytes) 2024-09-16T13:17:30.114 INFO:teuthology.orchestra.run.smithi098.stdout:2120 write 0x2db2ee3 thru 0x2dc06b3 (0xd7d1 bytes) 2024-09-16T13:17:30.115 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:30.107+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:30.126 INFO:teuthology.orchestra.run.smithi098.stdout:2122 writesame 0xbb1c03 thru 0xbb6246 data_size 0x6(0x4644 bytes) 2024-09-16T13:17:30.134 INFO:teuthology.orchestra.run.smithi098.stdout:2123 writesame 0x553532b thru 0x553bdae data_size 0x22(0x6a84 bytes) 2024-09-16T13:17:30.143 INFO:teuthology.orchestra.run.smithi098.stdout:2124 read 0xe84918e thru 0xe855f3f (0xcdb2 bytes) 2024-09-16T13:17:30.144 INFO:teuthology.orchestra.run.smithi098.stdout:2125 punch from 0xe4cc119 to 0xe4cfbef, (0x3ad6 bytes) 2024-09-16T13:17:30.144 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.144 INFO:teuthology.orchestra.run.smithi098.stdout:2127 write 0x2a2e002 thru 0x2a30b46 (0x2b45 bytes) 2024-09-16T13:17:30.152 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.164 INFO:teuthology.orchestra.run.smithi098.stdout:2129 trunc from 0xea807fb to 0x741728d 2024-09-16T13:17:30.183 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:30.183 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:30.189 INFO:teuthology.orchestra.run.smithi098.stdout:2130 read 0x51c936 thru 0x52a006 (0xd6d1 bytes) 2024-09-16T13:17:30.189 INFO:teuthology.orchestra.run.smithi098.stdout:2131 read 0x2d0a469 thru 0x2d127e9 (0x8381 bytes) 2024-09-16T13:17:30.189 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.190 INFO:teuthology.orchestra.run.smithi098.stdout:2134 punch from 0x175c78c to 0x176281c, (0x6090 bytes) 2024-09-16T13:17:30.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:30.205 INFO:teuthology.orchestra.run.smithi098.stdout:2137 write 0xb72b64e thru 0xb734fa8 (0x995b bytes) 2024-09-16T13:17:30.216 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:30.207+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:30.219 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.234 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:17:30.235 INFO:teuthology.orchestra.run.smithi098.stdout:2139 writesame 0xe3fdf34 thru 0xe3fe2bd data_size 0x3(0x38a bytes) 2024-09-16T13:17:30.237 INFO:teuthology.orchestra.run.smithi098.stdout:2140 trunc from 0xe3fe2be to 0xd138114 2024-09-16T13:17:30.246 INFO:teuthology.orchestra.run.smithi098.stdout:2141 write 0x73883f5 thru 0x738b16e (0x2d7a bytes) 2024-09-16T13:17:30.256 INFO:teuthology.orchestra.run.smithi098.stdout:2143 read 0x39486ea thru 0x3957d83 (0xf69a bytes) 2024-09-16T13:17:30.257 INFO:teuthology.orchestra.run.smithi098.stdout:2144 write 0xa19b713 thru 0xa1a836f (0xcc5d bytes) 2024-09-16T13:17:30.277 INFO:teuthology.orchestra.run.smithi098.stdout:2145 trunc from 0xd138114 to 0x88da680 2024-09-16T13:17:30.296 INFO:teuthology.orchestra.run.smithi098.stdout:2146 read 0x730e7e0 thru 0x7319724 (0xaf45 bytes) 2024-09-16T13:17:30.298 INFO:teuthology.orchestra.run.smithi098.stdout:2147 trunc from 0x88da680 to 0x9d9c9ad 2024-09-16T13:17:30.309 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:30.309 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:30.316 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:30.312+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:30.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:30.326 INFO:teuthology.orchestra.run.smithi098.stdout:2149 writesame 0xd26e9a9 thru 0xd277d81 data_size 0x1(0x93d9 bytes) 2024-09-16T13:17:30.340 INFO:teuthology.orchestra.run.smithi098.stdout:2150 write 0xe5f03f8 thru 0xe5f2ab9 (0x26c2 bytes) 2024-09-16T13:17:30.347 INFO:teuthology.orchestra.run.smithi098.stdout:2151 read 0x3259009 thru 0x325fcd3 (0x6ccb bytes) 2024-09-16T13:17:30.347 INFO:teuthology.orchestra.run.smithi098.stdout:2152 read 0x5c490ae thru 0x5c58003 (0xef56 bytes) 2024-09-16T13:17:30.347 INFO:teuthology.orchestra.run.smithi098.stdout:2153 punch from 0xda5a7c0 to 0xda66b74, (0xc3b4 bytes) 2024-09-16T13:17:30.347 INFO:teuthology.orchestra.run.smithi098.stdout:2154 read 0x13ee263 thru 0x13fdbc7 (0xf965 bytes) 2024-09-16T13:17:30.347 INFO:teuthology.orchestra.run.smithi098.stdout:2156 write 0x6de09cd thru 0x6dee621 (0xdc55 bytes) 2024-09-16T13:17:30.356 INFO:teuthology.orchestra.run.smithi098.stdout:2157 write 0x549b888 thru 0x549fdd3 (0x454c bytes) 2024-09-16T13:17:30.372 INFO:teuthology.orchestra.run.smithi098.stdout:2158 trunc from 0xe5f2aba to 0x9a0cddb 2024-09-16T13:17:30.390 INFO:teuthology.orchestra.run.smithi098.stdout:2159 read 0x21d28f4 thru 0x21d9726 (0x6e33 bytes) 2024-09-16T13:17:30.390 INFO:teuthology.orchestra.run.smithi098.stdout:2160 read 0x8b0ed8d thru 0x8b1b828 (0xca9c bytes) 2024-09-16T13:17:30.399 INFO:teuthology.orchestra.run.smithi098.stdout:2161 trunc from 0x9a0cddb to 0x4a85cd6 2024-09-16T13:17:30.417 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:30.412+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:30.430 INFO:teuthology.orchestra.run.smithi098.stdout:2162 write 0x4d42970 thru 0x4d4324b (0x8dc bytes) 2024-09-16T13:17:30.435 INFO:teuthology.orchestra.run.smithi098.stdout:2163 punch from 0x41874a8 to 0x4194bf4, (0xd74c bytes) 2024-09-16T13:17:30.435 INFO:teuthology.orchestra.run.smithi098.stdout:2164 read 0x21e5857 thru 0x21f4671 (0xee1b bytes) 2024-09-16T13:17:30.436 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:30.436 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:30.448 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:30.459 INFO:teuthology.orchestra.run.smithi098.stdout:2165 write 0xc1ad399 thru 0xc1b9962 (0xc5ca bytes) 2024-09-16T13:17:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:2166 read 0x2286a63 thru 0x2294229 (0xd7c7 bytes) 2024-09-16T13:17:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:2168 punch from 0x22d5390 to 0x22da348, (0x4fb8 bytes) 2024-09-16T13:17:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:2169 read 0x4b20979 thru 0x4b30827 (0xfeaf bytes) 2024-09-16T13:17:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:2170 write 0xa030e7e thru 0xa03fada (0xec5d bytes) 2024-09-16T13:17:30.484 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.485 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.485 INFO:teuthology.orchestra.run.smithi098.stdout:2173 read 0x545019b thru 0x545076f (0x5d5 bytes) 2024-09-16T13:17:30.485 INFO:teuthology.orchestra.run.smithi098.stdout:2175 read 0xf5d075 thru 0xf61657 (0x45e3 bytes) 2024-09-16T13:17:30.485 INFO:teuthology.orchestra.run.smithi098.stdout:2176 write 0x931a238 thru 0x9325a0c (0xb7d5 bytes) 2024-09-16T13:17:30.496 INFO:teuthology.orchestra.run.smithi098.stdout:2177 writesame 0xafb8f69 thru 0xafbc1c6 data_size 0x15(0x325e bytes) 2024-09-16T13:17:30.519 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:30.511+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:30.521 INFO:teuthology.orchestra.run.smithi098.stdout:2178 trunc from 0xc1b9963 to 0x20e0a2b 2024-09-16T13:17:30.560 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:30.560 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:30.564 INFO:teuthology.orchestra.run.smithi098.stdout:2179 write 0x762ecbd thru 0x7635544 (0x6888 bytes) 2024-09-16T13:17:30.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:30.581 INFO:teuthology.orchestra.run.smithi098.stdout:2180 writesame 0x82d1d95 thru 0x82d6745 data_size 0x37(0x49b1 bytes) 2024-09-16T13:17:30.582 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.582 INFO:teuthology.orchestra.run.smithi098.stdout:2182 write 0x11f85 thru 0x1eabd (0xcb39 bytes) 2024-09-16T13:17:30.584 INFO:teuthology.orchestra.run.smithi098.stdout:2183 punch from 0x9ab6a4 to 0x9ba2fd, (0xec59 bytes) 2024-09-16T13:17:30.596 INFO:teuthology.orchestra.run.smithi098.stdout:2184 write 0x9d970a7 thru 0x9d9d11f (0x6079 bytes) 2024-09-16T13:17:30.608 INFO:teuthology.orchestra.run.smithi098.stdout:2185 write 0x775a7ca thru 0x775aee9 (0x720 bytes) 2024-09-16T13:17:30.614 INFO:teuthology.orchestra.run.smithi098.stdout:2186 write 0x5222341 thru 0x5230dc6 (0xea86 bytes) 2024-09-16T13:17:30.620 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:30.616+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:30.626 INFO:teuthology.orchestra.run.smithi098.stdout:2187 read 0x65d2b6c thru 0x65e115a (0xe5ef bytes) 2024-09-16T13:17:30.628 INFO:teuthology.orchestra.run.smithi098.stdout:2188 punch from 0x1494aca to 0x14a0113, (0xb649 bytes) 2024-09-16T13:17:30.638 INFO:teuthology.orchestra.run.smithi098.stdout:2193 trunc from 0x9d9d120 to 0x2f5824f 2024-09-16T13:17:30.662 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.662 INFO:teuthology.orchestra.run.smithi098.stdout:2196 writesame 0x9418fb thru 0x9444f8 data_size 0x3(0x2bfe bytes) 2024-09-16T13:17:30.673 INFO:teuthology.orchestra.run.smithi098.stdout:2197 write 0x551bf4 thru 0x55eb02 (0xcf0f bytes) 2024-09-16T13:17:30.686 INFO:teuthology.orchestra.run.smithi098.stdout:2198 trunc from 0x2f5824f to 0x49a81bf 2024-09-16T13:17:30.691 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:30.691 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:30.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:30.719 INFO:teuthology.orchestra.run.smithi098.stdout:2200 write 0xcd61559 thru 0xcd6dfac (0xca54 bytes) 2024-09-16T13:17:30.721 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:30.716+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:30.737 INFO:teuthology.orchestra.run.smithi098.stdout:2201 trunc from 0xcd6dfad to 0xb19dd0c 2024-09-16T13:17:30.770 INFO:teuthology.orchestra.run.smithi098.stdout:2202 write 0xdceb46a thru 0xdcf4ea7 (0x9a3e bytes) 2024-09-16T13:17:30.784 INFO:teuthology.orchestra.run.smithi098.stdout:2204 punch from 0x7a7cbc5 to 0x7a7f983, (0x2dbe bytes) 2024-09-16T13:17:30.784 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:30.804 INFO:teuthology.orchestra.run.smithi098.stdout:2207 trunc from 0xdcf4ea8 to 0xfbede5 2024-09-16T13:17:30.821 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:30.816+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:30.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:30.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:30.839 INFO:teuthology.orchestra.run.smithi098.stdout:2208 write 0x1aa82ad thru 0x1ab7cae (0xfa02 bytes) 2024-09-16T13:17:30.851 INFO:teuthology.orchestra.run.smithi098.stdout:2209 punch from 0x18a4c07 to 0x18b0fbb, (0xc3b4 bytes) 2024-09-16T13:17:30.863 INFO:teuthology.orchestra.run.smithi098.stdout:2210 writesame 0x3a6f832 thru 0x3a7ad3a data_size 0xd(0xb509 bytes) 2024-09-16T13:17:30.876 INFO:teuthology.orchestra.run.smithi098.stdout:2211 trunc from 0x3a7ad3b to 0xb171b8b 2024-09-16T13:17:30.887 INFO:teuthology.orchestra.run.smithi098.stdout:2214 read 0x9d4abb0 thru 0x9d54fef (0xa440 bytes) 2024-09-16T13:17:30.888 INFO:teuthology.orchestra.run.smithi098.stdout:2216 trunc from 0xb171b8b to 0xa75758c 2024-09-16T13:17:30.903 INFO:teuthology.orchestra.run.smithi098.stdout:2217 write 0x705aaa8 thru 0x705b4b2 (0xa0b bytes) 2024-09-16T13:17:30.916 INFO:teuthology.orchestra.run.smithi098.stdout:2218 trunc from 0xa75758c to 0xeb4c545 2024-09-16T13:17:30.923 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:30.915+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:30.926 INFO:teuthology.orchestra.run.smithi098.stdout:2220 punch from 0x28403d0 to 0x2844de7, (0x4a17 bytes) 2024-09-16T13:17:30.946 INFO:teuthology.orchestra.run.smithi098.stdout:2222 trunc from 0xeb4c545 to 0x26ecc7c 2024-09-16T13:17:30.957 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:30.957 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:30.965 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:30.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:30.977 INFO:teuthology.orchestra.run.smithi098.stdout:2223 trunc from 0x26ecc7c to 0xa3add65 2024-09-16T13:17:30.984 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:31.004 INFO:teuthology.orchestra.run.smithi098.stdout:2224 write 0xea7a4b5 thru 0xea7e03e (0x3b8a bytes) 2024-09-16T13:17:31.008 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:31.014 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.019 INFO:teuthology.orchestra.run.smithi098.stdout:2226 trunc from 0xea7e03f to 0xb9bfbec 2024-09-16T13:17:31.023 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:31.020+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:31.025 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:31.040 INFO:teuthology.orchestra.run.smithi098.stdout:2227 punch from 0x5bd512e to 0x5bd53a1, (0x273 bytes) 2024-09-16T13:17:31.040 INFO:teuthology.orchestra.run.smithi098.stdout:2229 write 0x47ab62 thru 0x47c28f (0x172e bytes) 2024-09-16T13:17:31.043 INFO:teuthology.orchestra.run.smithi098.stdout:2230 writesame 0x27faa65 thru 0x2805273 data_size 0x1(0xa80f bytes) 2024-09-16T13:17:31.050 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:31.077 INFO:teuthology.orchestra.run.smithi098.stdout:2231 writesame 0x58882e3 thru 0x588fddc data_size 0x42(0x7afa bytes) 2024-09-16T13:17:31.077 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:31.085 INFO:teuthology.orchestra.run.smithi098.stdout:2232 writesame 0x489a6ea thru 0x48a73d6 data_size 0x1d(0xcced bytes) 2024-09-16T13:17:31.088 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:31.089 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:31.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:31.089 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:31.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.089 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5101 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.089 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:31.089 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.304630896999999, 2024-09-16T13:17:31.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595248, 2024-09-16T13:17:31.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.089 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5101 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777333+0000", 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1949999999999996e-06 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777342+0000", 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6950000000000006e-06 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777348+0000", 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785241+0000", 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078929329999999995 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785391+0000", 2024-09-16T13:17:31.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150113 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785969+0000", 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057822899999999998 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789745+0000", 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037756529999999999 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793823+0000", 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040786299999999998 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793880+0000", 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7157999999999997e-05 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793924+0000", 2024-09-16T13:17:31.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3407e-05 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5102 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.304583605000001, 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016663710000000002, 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.096 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5102 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777377+0000", 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4600000000000003e-07 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777381+0000", 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5290000000000002e-06 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777393+0000", 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2001000000000001e-05 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777401+0000", 2024-09-16T13:17:31.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3869999999999997e-06 2024-09-16T13:17:31.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786753+0000", 2024-09-16T13:17:31.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093523649999999996 2024-09-16T13:17:31.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786860+0000", 2024-09-16T13:17:31.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106921 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787286+0000", 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042651800000000003 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793815+0000", 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065290579999999999 2024-09-16T13:17:31.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793932+0000", 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116688 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794002+0000", 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0313000000000003e-05 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794040+0000", 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7083999999999999e-05 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5103 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.304522719000001, 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017505544000000001, 2024-09-16T13:17:31.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.104 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5103 2024-09-16T13:17:31.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.104 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777438+0000", 2024-09-16T13:17:31.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.78e-07 2024-09-16T13:17:31.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777442+0000", 2024-09-16T13:17:31.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5950000000000001e-06 2024-09-16T13:17:31.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777453+0000", 2024-09-16T13:17:31.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0372e-05 2024-09-16T13:17:31.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777458+0000", 2024-09-16T13:17:31.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6559999999999997e-06 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788034+0000", 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010575753 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788111+0000", 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7038000000000003e-05 2024-09-16T13:17:31.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788463+0000", 2024-09-16T13:17:31.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000352028 2024-09-16T13:17:31.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794263+0000", 2024-09-16T13:17:31.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005800342 2024-09-16T13:17:31.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794837+0000", 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057334300000000003 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794897+0000", 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0690000000000003e-05 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794942+0000", 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4848999999999997e-05 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5104 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:31.111 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.304478311, 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017694142999999999, 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5104 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:31.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:31.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777482+0000", 2024-09-16T13:17:31.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:17:31.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777487+0000", 2024-09-16T13:17:31.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6639999999999997e-06 2024-09-16T13:17:31.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777496+0000", 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2890000000000003e-06 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777501+0000", 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2730000000000002e-06 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788951+0000", 2024-09-16T13:17:31.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011449429000000001 2024-09-16T13:17:31.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789007+0000", 2024-09-16T13:17:31.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5643999999999998e-05 2024-09-16T13:17:31.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789205+0000", 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019820800000000001 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794268+0000", 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050636850000000001 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795094+0000", 2024-09-16T13:17:31.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082569700000000002 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795139+0000", 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5055000000000001e-05 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795175+0000", 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6149000000000002e-05 2024-09-16T13:17:31.118 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5105 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.304415970000001, 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017729482000000001, 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.119 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5105 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777544+0000", 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1100000000000004e-07 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777549+0000", 2024-09-16T13:17:31.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777561+0000", 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1524e-05 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777566+0000", 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.558e-06 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789781+0000", 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012214981 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789870+0000", 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9171000000000004e-05 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790313+0000", 2024-09-16T13:17:31.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044255299999999999 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794273+0000", 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039596129999999998 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795213+0000", 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094086099999999998 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795239+0000", 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5324e-05 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795273+0000", 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4198e-05 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5106 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.304370826, 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018621867, 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.126 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5106 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777590+0000", 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777594+0000", 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6750000000000001e-06 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777604+0000", 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0076e-05 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777610+0000", 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7529999999999998e-06 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790893+0000", 2024-09-16T13:17:31.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013282845 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790999+0000", 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106289 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791493+0000", 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049357800000000005 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794662+0000", 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031691409999999999 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796146+0000", 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00148375 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796176+0000", 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0349e-05 2024-09-16T13:17:31.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796211+0000", 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4530000000000003e-05 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5107 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.304339564999999, 2024-09-16T13:17:31.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018745067000000001, 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5107 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:31.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777621+0000", 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-07 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777624+0000", 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7950000000000001e-06 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777633+0000", 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5979999999999997e-06 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777638+0000", 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.989e-06 2024-09-16T13:17:31.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792024+0000", 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014385692 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792104+0000", 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0567000000000003e-05 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792340+0000", 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023624200000000001 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794667+0000", 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023269990000000002 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796304+0000", 2024-09-16T13:17:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001636981 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796331+0000", 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6492999999999999e-05 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796365+0000", 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4102e-05 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5108 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.304285736000001, 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019090617000000001, 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5108 2024-09-16T13:17:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777677+0000", 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7879999999999998e-06 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777686+0000", 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7679999999999997e-06 2024-09-16T13:17:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777689+0000", 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.123e-06 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792751+0000", 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01506158 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792855+0000", 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010456 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793305+0000", 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045006100000000003 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794671+0000", 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001366304 2024-09-16T13:17:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796393+0000", 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001721272 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796733+0000", 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000340611 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796764+0000", 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.099e-05 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5208 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.202526285000001, 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014882177999999999, 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5208 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879434+0000", 2024-09-16T13:17:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.322e-06 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879443+0000", 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879459+0000", 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6133999999999998e-05 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879467+0000", 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2400000000000007e-06 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888339+0000", 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088721619999999994 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888461+0000", 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121341 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888819+0000", 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035782799999999998 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891271+0000", 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002452872 2024-09-16T13:17:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894224+0000", 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029522910000000001 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894275+0000", 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1647999999999997e-05 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894315+0000", 2024-09-16T13:17:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9993999999999997e-05 2024-09-16T13:17:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5210 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.201971097000001, 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01466423, 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5210 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879989+0000", 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8800000000000001e-07 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879995+0000", 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880017+0000", 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2101000000000001e-05 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880024+0000", 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9240000000000001e-06 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890250+0000", 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010226012 2024-09-16T13:17:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890344+0000", 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.446e-05 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890770+0000", 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042575099999999998 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894344+0000", 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003574163 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894549+0000", 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020477599999999999 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894606+0000", 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6891000000000001e-05 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894653+0000", 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6867999999999999e-05 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5211 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.201927034000001, 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014800604, 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5211 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880033+0000", 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.06e-07 2024-09-16T13:17:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880039+0000", 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880052+0000", 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-05 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880059+0000", 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3090000000000001e-06 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891316+0000", 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011257118999999999 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891374+0000", 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8020999999999999e-05 2024-09-16T13:17:31.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891577+0000", 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020371699999999999 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894348+0000", 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027708300000000002 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894726+0000", 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037819399999999999 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894784+0000", 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7667999999999997e-05 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894833+0000", 2024-09-16T13:17:31.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8936e-05 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5212 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.201571659999999, 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014645704000000001, 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.166 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5212 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880389+0000", 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1680000000000001e-06 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880395+0000", 2024-09-16T13:17:31.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880408+0000", 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.308e-05 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880414+0000", 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.105e-06 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892037+0000", 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011622402 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892116+0000", 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9939999999999997e-05 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892597+0000", 2024-09-16T13:17:31.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048076500000000002 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894492+0000", 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001895246 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894921+0000", 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000428114 2024-09-16T13:17:31.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894985+0000", 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4777999999999995e-05 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895034+0000", 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8133999999999997e-05 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5213 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.201438509999999, 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014632608, 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.173 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5213 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880522+0000", 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2600000000000001e-07 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880527+0000", 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880538+0000", 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917999999999999e-05 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880543+0000", 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.361e-06 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893069+0000", 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012525497999999999 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893152+0000", 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2805000000000004e-05 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893635+0000", 2024-09-16T13:17:31.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048315100000000002 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894852+0000", 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001216806 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895075+0000", 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022300299999999999 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895122+0000", 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7077999999999999e-05 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895154+0000", 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1919999999999999e-05 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.179 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5214 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.201283859, 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017785977000000001, 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5214 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880681+0000", 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880689+0000", 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3610000000000001e-06 2024-09-16T13:17:31.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880694+0000", 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5809999999999998e-06 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895186+0000", 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014492430000000001 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895311+0000", 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012518300000000001 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895760+0000", 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044864900000000002 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898337+0000", 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025765229999999998 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898354+0000", 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7538999999999999e-05 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898406+0000", 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2054e-05 2024-09-16T13:17:31.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898462+0000", 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5404e-05 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5215 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.186 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.201019025000001, 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018191319000000001, 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5215 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880941+0000", 2024-09-16T13:17:31.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-16T13:17:31.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880946+0000", 2024-09-16T13:17:31.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3710000000000002e-06 2024-09-16T13:17:31.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880955+0000", 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1230000000000006e-06 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880959+0000", 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896782+0000", 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015822716000000001 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896876+0000", 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3667999999999998e-05 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.897289+0000", 2024-09-16T13:17:31.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041357399999999999 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898474+0000", 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011847209999999999 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899036+0000", 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056215699999999998 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899090+0000", 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3662999999999998e-05 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899132+0000", 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1814000000000003e-05 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5282 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.132346910999999, 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014924988, 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5282 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:31.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.53e-07 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949635+0000", 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2422999999999999e-05 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949641+0000", 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.896e-06 2024-09-16T13:17:31.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949644+0000", 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5270000000000001e-06 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959656+0000", 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010012378000000001 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959747+0000", 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1273000000000005e-05 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960045+0000", 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029733 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963196+0000", 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031512620000000002 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964453+0000", 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001257357 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964502+0000", 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9044e-05 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964538+0000", 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5244999999999999e-05 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5283 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.132311757, 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015065762, 2024-09-16T13:17:31.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5283 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949650+0000", 2024-09-16T13:17:31.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.736e-06 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949653+0000", 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2969999999999999e-06 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949655+0000", 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1890000000000001e-06 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960643+0000", 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010988015 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960732+0000", 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9097000000000004e-05 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961195+0000", 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000462292 2024-09-16T13:17:31.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963208+0000", 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002013641 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964585+0000", 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013765769999999999 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964669+0000", 2024-09-16T13:17:31.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3863000000000007e-05 2024-09-16T13:17:31.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964713+0000", 2024-09-16T13:17:31.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4706999999999997e-05 2024-09-16T13:17:31.206 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.206 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.207 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.207 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5284 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.207 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:31.207 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.132301888000001, 2024-09-16T13:17:31.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015152802, 2024-09-16T13:17:31.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.207 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5284 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5899999999999998e-07 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949659+0000", 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.601e-06 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949672+0000", 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2364999999999999e-05 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949674+0000", 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.210 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961869+0000", 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012194903999999999 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961971+0000", 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102154 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.962502+0000", 2024-09-16T13:17:31.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00053037500000000003 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964751+0000", 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022487800000000001 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964755+0000", 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964789+0000", 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4054000000000003e-05 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964810+0000", 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0922000000000001e-05 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5285 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.132282969999999, 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015553022, 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5285 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.214 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949677+0000", 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6899999999999999e-07 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949679+0000", 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0090000000000002e-06 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949682+0000", 2024-09-16T13:17:31.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2550000000000002e-06 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949685+0000", 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.621e-06 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963409+0000", 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013724485999999999 2024-09-16T13:17:31.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963497+0000", 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7560999999999997e-05 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963933+0000", 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043592700000000002 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.218 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964820+0000", 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088772299999999998 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965165+0000", 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000344426 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965201+0000", 2024-09-16T13:17:31.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5967000000000003e-05 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965230+0000", 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11223 3.8 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 33927~20122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:31.220 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.621110639999999, 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019333487, 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11223 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:31.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460851+0000", 2024-09-16T13:17:31.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5730000000000002e-06 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464824+0000", 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039725730000000001 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464861+0000", 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7233000000000001e-05 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464872+0000", 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209000000000001e-05 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464987+0000", 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011423 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465188+0000", 2024-09-16T13:17:31.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020091600000000001 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465929+0000", 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00074102 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.475697+0000", 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009768367 2024-09-16T13:17:31.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480068+0000", 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043713290000000002 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480143+0000", 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5000999999999996e-05 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:31.227 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480182+0000", 2024-09-16T13:17:31.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9035999999999997e-05 2024-09-16T13:17:31.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.227 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:31.228 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.228 INFO:teuthology.orchestra.run.smithi098.stdout:2234 punch from 0x1280ca1 to 0x1281e88, (0x11e7 bytes) 2024-09-16T13:17:31.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:31.230 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:31.230 INFO:teuthology.orchestra.run.smithi098.stdout:2235 trunc from 0xb9bfbec to 0x953a755 2024-09-16T13:17:31.231 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.231 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.231 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.231 INFO:teuthology.orchestra.run.smithi098.stdout:2239 write 0x52ad104 thru 0x52bb50a (0xe407 bytes) 2024-09-16T13:17:31.231 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:31.232 INFO:teuthology.orchestra.run.smithi098.stdout:2240 trunc from 0x953a755 to 0xcc47c5b 2024-09-16T13:17:31.234 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:31.219+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:31.234 INFO:teuthology.orchestra.run.smithi098.stdout:2241 trunc from 0xcc47c5b to 0x20de58b 2024-09-16T13:17:31.235 INFO:teuthology.orchestra.run.smithi098.stdout:2242 trunc from 0x20de58b to 0x5739250 2024-09-16T13:17:31.235 INFO:teuthology.orchestra.run.smithi098.stdout:2243 read 0x3a7c53d thru 0x3a86891 (0xa355 bytes) 2024-09-16T13:17:31.235 INFO:teuthology.orchestra.run.smithi098.stdout:2245 read 0xc15027 thru 0xc1ea6a (0x9a44 bytes) 2024-09-16T13:17:31.235 INFO:teuthology.orchestra.run.smithi098.stdout:2246 write 0x529c984 thru 0x529fba8 (0x3225 bytes) 2024-09-16T13:17:31.235 INFO:teuthology.orchestra.run.smithi098.stdout:2250 read 0x3b7157e thru 0x3b71fd0 (0xa53 bytes) 2024-09-16T13:17:31.235 INFO:teuthology.orchestra.run.smithi098.stdout:2251 write 0xe13abfa thru 0xe13d4b5 (0x28bc bytes) 2024-09-16T13:17:31.243 INFO:teuthology.orchestra.run.smithi098.stdout:2252 read 0x9dbd307 thru 0x9dc3e4c (0x6b46 bytes) 2024-09-16T13:17:31.243 INFO:teuthology.orchestra.run.smithi098.stdout:2254 write 0x7dd94ca thru 0x7de8141 (0xec78 bytes) 2024-09-16T13:17:31.254 INFO:teuthology.orchestra.run.smithi098.stdout:2255 write 0x1dfbd44 thru 0x1e081e1 (0xc49e bytes) 2024-09-16T13:17:31.265 INFO:teuthology.orchestra.run.smithi098.stdout:2256 write 0xd2d0758 thru 0xd2dfcdd (0xf586 bytes) 2024-09-16T13:17:31.274 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.298 INFO:teuthology.orchestra.run.smithi098.stdout:2258 trunc from 0xe13d4b6 to 0x745287 2024-09-16T13:17:31.311 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:31.311 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:31.312 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-16T13:17:31.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T13:17:31.324 INFO:teuthology.orchestra.run.smithi098.stdout:2259 read 0xf8454 thru 0xfae87 (0x2a34 bytes) 2024-09-16T13:17:31.325 INFO:teuthology.orchestra.run.smithi098.stdout:2260 read 0x31941d thru 0x32603c (0xcc20 bytes) 2024-09-16T13:17:31.329 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:31.323+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:31.346 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:31.346 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:31.346 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:31.346 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:31.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:31.356 INFO:teuthology.orchestra.run.smithi098.stdout:2261 write 0xa454686 thru 0xa4583cd (0x3d48 bytes) 2024-09-16T13:17:31.365 INFO:teuthology.orchestra.run.smithi098.stdout:2262 write 0x2fe78f9 thru 0x2febb75 (0x427d bytes) 2024-09-16T13:17:31.374 INFO:teuthology.orchestra.run.smithi098.stdout:2263 write 0xc61dba thru 0xc6796d (0x5bb4 bytes) 2024-09-16T13:17:31.380 INFO:teuthology.orchestra.run.smithi098.stdout:2264 read 0x7bd84f0 thru 0x7be2290 (0x9da1 bytes) 2024-09-16T13:17:31.380 INFO:teuthology.orchestra.run.smithi098.stdout:2265 writesame 0x48ba135 thru 0x48bb785 data_size 0x1(0x1651 bytes) 2024-09-16T13:17:31.390 INFO:teuthology.orchestra.run.smithi098.stdout:2266 writesame 0x7084ba thru 0x713cc6 data_size 0x35(0xb80d bytes) 2024-09-16T13:17:31.399 INFO:teuthology.orchestra.run.smithi098.stdout:2268 punch from 0x75b5165 to 0x75bd238, (0x80d3 bytes) 2024-09-16T13:17:31.399 INFO:teuthology.orchestra.run.smithi098.stdout:2270 punch from 0x9c92e4b to 0x9c9e249, (0xb3fe bytes) 2024-09-16T13:17:31.399 INFO:teuthology.orchestra.run.smithi098.stdout:2272 write 0x4684a9c thru 0x468a7c9 (0x5d2e bytes) 2024-09-16T13:17:31.404 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.404 INFO:teuthology.orchestra.run.smithi098.stdout:2274 trunc from 0xa4583ce to 0xa3390b6 2024-09-16T13:17:31.416 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.428 INFO:teuthology.orchestra.run.smithi098.stdout:2276 write 0xbe90aed thru 0xbe97870 (0x6d84 bytes) 2024-09-16T13:17:31.430 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:31.423+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:31.436 INFO:teuthology.orchestra.run.smithi098.stdout:2277 punch from 0x5827d7 to 0x58319b, (0x9c4 bytes) 2024-09-16T13:17:31.436 INFO:teuthology.orchestra.run.smithi098.stdout:2278 write 0x6cd4924 thru 0x6cd8b97 (0x4274 bytes) 2024-09-16T13:17:31.443 INFO:teuthology.orchestra.run.smithi098.stdout:2279 write 0x91a424e thru 0x91a5934 (0x16e7 bytes) 2024-09-16T13:17:31.448 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.449 INFO:teuthology.orchestra.run.smithi098.stdout:2281 trunc from 0xbe97871 to 0xb5920dd 2024-09-16T13:17:31.457 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:31.457 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:31.457 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:31.457 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:31.457 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:31.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:31.481 INFO:teuthology.orchestra.run.smithi098.stdout:2282 trunc from 0xb5920dd to 0x5f977b8 2024-09-16T13:17:31.500 INFO:teuthology.orchestra.run.smithi098.stdout:2283 read 0x5531114 thru 0x553499b (0x3888 bytes) 2024-09-16T13:17:31.500 INFO:teuthology.orchestra.run.smithi098.stdout:2285 writesame 0x198dda1 thru 0x198e63c data_size 0x1(0x89c bytes) 2024-09-16T13:17:31.526 INFO:teuthology.orchestra.run.smithi098.stdout:2286 write 0xa33c155 thru 0xa33c6b0 (0x55c bytes) 2024-09-16T13:17:31.530 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:31.524+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:31.533 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.533 INFO:teuthology.orchestra.run.smithi098.stdout:2292 write 0x1e67727 thru 0x1e6e57d (0x6e57 bytes) 2024-09-16T13:17:31.544 INFO:teuthology.orchestra.run.smithi098.stdout:2293 read 0x5e134a9 thru 0x5e21ff1 (0xeb49 bytes) 2024-09-16T13:17:31.544 INFO:teuthology.orchestra.run.smithi098.stdout:2294 writesame 0x21e14bc thru 0x21e78eb data_size 0x1c(0x6430 bytes) 2024-09-16T13:17:31.551 INFO:teuthology.orchestra.run.smithi098.stdout:2295 write 0x80eff4a thru 0x80ff607 (0xf6be bytes) 2024-09-16T13:17:31.566 INFO:teuthology.orchestra.run.smithi098.stdout:2296 punch from 0x62fe94c to 0x6306edd, (0x8591 bytes) 2024-09-16T13:17:31.570 INFO:teuthology.orchestra.run.smithi098.stdout:2297 trunc from 0xa33c6b1 to 0xc363162 2024-09-16T13:17:31.578 INFO:teuthology.orchestra.run.smithi098.stdout:2298 punch from 0x688a1d0 to 0x688b694, (0x14c4 bytes) 2024-09-16T13:17:31.578 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.578 INFO:teuthology.orchestra.run.smithi098.stdout:2300 writesame 0x7677eb3 thru 0x7683a90 data_size 0x8b(0xbbde bytes) 2024-09-16T13:17:31.590 INFO:teuthology.orchestra.run.smithi098.stdout:2302 read 0xb25b0fb thru 0xb25dd8f (0x2c95 bytes) 2024-09-16T13:17:31.590 INFO:teuthology.orchestra.run.smithi098.stdout:2303 read 0xa08a11e thru 0xa08d3f0 (0x32d3 bytes) 2024-09-16T13:17:31.590 INFO:teuthology.orchestra.run.smithi098.stdout:2304 read 0x66f991 thru 0x679dd2 (0xa442 bytes) 2024-09-16T13:17:31.591 INFO:teuthology.orchestra.run.smithi098.stdout:2305 punch from 0x10d178c to 0x10da87f, (0x90f3 bytes) 2024-09-16T13:17:31.591 INFO:teuthology.orchestra.run.smithi098.stdout:2307 write 0x72fc4d7 thru 0x73062d3 (0x9dfd bytes) 2024-09-16T13:17:31.602 INFO:teuthology.orchestra.run.smithi098.stdout:2308 read 0x85a156 thru 0x864258 (0xa103 bytes) 2024-09-16T13:17:31.603 INFO:teuthology.orchestra.run.smithi098.stdout:2310 read 0xada74f2 thru 0xadaabd5 (0x36e4 bytes) 2024-09-16T13:17:31.606 INFO:teuthology.orchestra.run.smithi098.stdout:2311 trunc from 0xc363162 to 0xdb4403e 2024-09-16T13:17:31.615 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:31.615 INFO:teuthology.orchestra.run.smithi098.stdout:2314 writesame 0x204bf26 thru 0x204edc5 data_size 0x10(0x2ea0 bytes) 2024-09-16T13:17:31.623 INFO:teuthology.orchestra.run.smithi098.stdout:2316 write 0x3f49ef4 thru 0x3f526bb (0x87c8 bytes) 2024-09-16T13:17:31.629 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:31.630 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:31.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:31.630 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:31.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.630 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8369 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.630 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:31.630 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.567168017, 2024-09-16T13:17:31.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020058919000000001, 2024-09-16T13:17:31.630 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.630 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8369 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056335+0000", 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056338+0000", 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056344+0000", 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8150000000000002e-06 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056348+0000", 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.070951+0000", 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014603055 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071060+0000", 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109462 2024-09-16T13:17:31.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071508+0000", 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044824200000000001 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074304+0000", 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027951719999999998 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076331+0000", 2024-09-16T13:17:31.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020276529999999999 2024-09-16T13:17:31.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076369+0000", 2024-09-16T13:17:31.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7557e-05 2024-09-16T13:17:31.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076393+0000", 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4445999999999999e-05 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8370 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.567150698999999, 2024-09-16T13:17:31.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021240851000000002, 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8370 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-16T13:17:31.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056355+0000", 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.667e-06 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056360+0000", 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6620000000000002e-06 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056366+0000", 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072672+0000", 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016306157000000002 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072746+0000", 2024-09-16T13:17:31.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3648999999999999e-05 2024-09-16T13:17:31.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.073364+0000", 2024-09-16T13:17:31.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061885499999999997 2024-09-16T13:17:31.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076445+0000", 2024-09-16T13:17:31.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003080396 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077482+0000", 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001036709 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077535+0000", 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3211000000000003e-05 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077592+0000", 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7664000000000002e-05 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8371 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.567121904, 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021386721000000001, 2024-09-16T13:17:31.645 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8371 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056381+0000", 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056385+0000", 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056391+0000", 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6959999999999999e-06 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056396+0000", 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2549999999999999e-06 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074355+0000", 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017959098999999999 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074503+0000", 2024-09-16T13:17:31.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014797200000000001 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074955+0000", 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045213300000000002 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077607+0000", 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026519500000000001 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077660+0000", 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077725+0000", 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4991000000000002e-05 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077767+0000", 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8372 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.567100882999998, 2024-09-16T13:17:31.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021557531000000001, 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:31.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056402+0000", 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1399999999999997e-07 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056405+0000", 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6520000000000002e-06 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056411+0000", 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2009999999999996e-06 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056415+0000", 2024-09-16T13:17:31.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.211e-06 2024-09-16T13:17:31.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075482+0000", 2024-09-16T13:17:31.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019067163000000002 2024-09-16T13:17:31.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075546+0000", 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3516000000000006e-05 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075844+0000", 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029810999999999998 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077784+0000", 2024-09-16T13:17:31.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019399350000000001 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077823+0000", 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9524000000000003e-05 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077917+0000", 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4090000000000002e-05 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077959+0000", 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1615000000000001e-05 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8373 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:31.659 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.567080539999999, 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022897421000000001, 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3200000000000005e-07 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056427+0000", 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-06 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056433+0000", 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056437+0000", 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076470+0000", 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020032239 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076524+0000", 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4475000000000003e-05 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076996+0000", 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000471905 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.078749+0000", 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001752539 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079217+0000", 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046819400000000001 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079276+0000", 2024-09-16T13:17:31.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9407000000000002e-05 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079319+0000", 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.666 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8398 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.537716748000001, 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017439282, 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8398 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.667 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085788+0000", 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4719999999999998e-06 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085793+0000", 2024-09-16T13:17:31.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6179999999999997e-06 2024-09-16T13:17:31.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085796+0000", 2024-09-16T13:17:31.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6680000000000001e-06 2024-09-16T13:17:31.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099514+0000", 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013717895000000001 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099873+0000", 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035970799999999998 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100269+0000", 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000395382 2024-09-16T13:17:31.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103068+0000", 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027992680000000002 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103114+0000", 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6189000000000001e-05 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103178+0000", 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3845999999999998e-05 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103225+0000", 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6888999999999998e-05 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8399 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.537675271000001, 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017551573000000001, 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8399 2024-09-16T13:17:31.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085828+0000", 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085830+0000", 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8609999999999998e-06 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085835+0000", 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.437e-06 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085837+0000", 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.571e-06 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100950+0000", 2024-09-16T13:17:31.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015112537000000001 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101047+0000", 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6945000000000001e-05 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101468+0000", 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000420794 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103315+0000", 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018474769999999999 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103321+0000", 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103356+0000", 2024-09-16T13:17:31.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4764000000000001e-05 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103379+0000", 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3102999999999999e-05 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.680 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8400 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.53766237, 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017960752, 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8400 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.53e-07 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085842+0000", 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2620000000000001e-06 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085847+0000", 2024-09-16T13:17:31.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2119999999999997e-06 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085849+0000", 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.204e-06 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101995+0000", 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016146384999999999 2024-09-16T13:17:31.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102100+0000", 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010482199999999999 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102466+0000", 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000365731 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103389+0000", 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092367199999999995 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103702+0000", 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031220400000000001 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103754+0000", 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2318999999999998e-05 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103801+0000", 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6687999999999998e-05 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8539 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.379530814999999, 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018486425000000001, 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8539 2024-09-16T13:17:31.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.689 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:31.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:31.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:31.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0500000000000003e-07 2024-09-16T13:17:31.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243975+0000", 2024-09-16T13:17:31.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800000000000002e-06 2024-09-16T13:17:31.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243982+0000", 2024-09-16T13:17:31.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243986+0000", 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4100000000000001e-06 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258523+0000", 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014536604999999999 2024-09-16T13:17:31.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258678+0000", 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155341 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.259218+0000", 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053930600000000003 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262325+0000", 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031076799999999998 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262367+0000", 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1480000000000003e-05 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262417+0000", 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0555e-05 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262458+0000", 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0556999999999998e-05 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8540 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.379511394000001, 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018906683000000001, 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8540 2024-09-16T13:17:31.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243992+0000", 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.75e-07 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243995+0000", 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1219999999999999e-06 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244001+0000", 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5010000000000001e-06 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244006+0000", 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999996e-06 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260137+0000", 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016131 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260200+0000", 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3782999999999996e-05 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260598+0000", 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039746600000000002 2024-09-16T13:17:31.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262331+0000", 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017327849999999999 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262786+0000", 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045546300000000003 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262851+0000", 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-05 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262898+0000", 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7098000000000002e-05 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.702 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8541 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.379492170999999, 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019366889000000002, 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.703 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.704 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8541 2024-09-16T13:17:31.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.704 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:31.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244011+0000", 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244014+0000", 2024-09-16T13:17:31.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.79e-06 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244020+0000", 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6370000000000002e-06 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244025+0000", 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3359999999999997e-06 2024-09-16T13:17:31.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261122+0000", 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017097161 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261228+0000", 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105989 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:31.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261475+0000", 2024-09-16T13:17:31.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024757000000000002 2024-09-16T13:17:31.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263022+0000", 2024-09-16T13:17:31.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015466760000000001 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263300+0000", 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027828100000000001 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263344+0000", 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4008999999999998e-05 2024-09-16T13:17:31.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263377+0000", 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2832e-05 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11776 3.1a 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:31.710 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.106089164, 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027432719000000001, 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11776 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517415+0000", 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5239999999999999e-06 2024-09-16T13:17:31.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526328+0000", 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089138059999999998 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526362+0000", 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3775000000000001e-05 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526379+0000", 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6835000000000001e-05 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526467+0000", 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7703999999999999e-05 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526653+0000", 2024-09-16T13:17:31.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186379 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.527624+0000", 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097080400000000002 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543313+0000", 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015689456000000001 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544696+0000", 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013822979999999999 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544784+0000", 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8419999999999997e-05 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544846+0000", 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1717999999999994e-05 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:11778 3.1 e40/23 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head v 40'57, mlcod=40'55)", 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.099605136999999, 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01745787, 2024-09-16T13:17:31.717 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523898+0000", 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-06 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533067+0000", 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091691120000000001 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533101+0000", 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3733999999999999e-05 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533115+0000", 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3410999999999999e-05 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533194+0000", 2024-09-16T13:17:31.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9827000000000007e-05 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533223+0000", 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8937000000000001e-05 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541297+0000", 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080739850000000005 2024-09-16T13:17:31.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541355+0000", 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7708999999999999e-05 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11777 3.1a 3:59cb7a3f:::rbd_object_map.112d21cb8818:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.097100226, 2024-09-16T13:17:31.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018649718999999999, 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11777 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:31.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526404+0000", 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.654e-06 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526409+0000", 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.882e-06 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526424+0000", 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5328999999999998e-05 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526428+0000", 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6219999999999998e-06 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542209+0000", 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015781765999999999 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542334+0000", 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125148 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542804+0000", 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046944899999999998 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543732+0000", 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092824699999999999 2024-09-16T13:17:31.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544929+0000", 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001196522 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545004+0000", 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4962999999999994e-05 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545052+0000", 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8136999999999997e-05 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11779 3.9 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.076578644, 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020806181999999999, 2024-09-16T13:17:31.729 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11779 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546926+0000", 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9769999999999999e-06 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552330+0000", 2024-09-16T13:17:31.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054043980000000004 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552353+0000", 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2875999999999999e-05 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552362+0000", 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7299999999999994e-06 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552447+0000", 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5620999999999999e-05 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552710+0000", 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026273300000000002 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.553446+0000", 2024-09-16T13:17:31.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073563299999999997 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.559002+0000", 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055564580000000002 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567615+0000", 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086126680000000004 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567689+0000", 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3839999999999998e-05 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567730+0000", 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1248000000000002e-05 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.735 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13736 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.3255074669999996, 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018229967999999999, 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13736 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.736 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2e-07 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297998+0000", 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.824e-06 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298004+0000", 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8730000000000004e-06 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298008+0000", 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312651+0000", 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014643351000000001 2024-09-16T13:17:31.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312740+0000", 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8466000000000003e-05 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313081+0000", 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034067899999999998 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315982+0000", 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002900943 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316051+0000", 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9029999999999995e-05 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316101+0000", 2024-09-16T13:17:31.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0680000000000003e-05 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316225+0000", 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012359099999999999 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13737 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:31.742 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.3254903470000006, 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018381208999999999, 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13737 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298015+0000", 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6869999999999999e-06 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298021+0000", 2024-09-16T13:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7799999999999997e-06 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298027+0000", 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.071e-06 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313829+0000", 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015801696 2024-09-16T13:17:31.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314019+0000", 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018995700000000001 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314379+0000", 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036036500000000002 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315988+0000", 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016084949999999999 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316290+0000", 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000302732 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316324+0000", 2024-09-16T13:17:31.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3937999999999999e-05 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316393+0000", 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8966000000000003e-05 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13738 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:31.749 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.3254713650000003, 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021400311000000002, 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13738 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298032+0000", 2024-09-16T13:17:31.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7400000000000003e-07 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298034+0000", 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5770000000000001e-06 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298040+0000", 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7010000000000001e-06 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298044+0000", 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2069999999999997e-06 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315033+0000", 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016988970999999999 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315106+0000", 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2723000000000002e-05 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315428+0000", 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032191099999999997 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319249+0000", 2024-09-16T13:17:31.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003821166 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319313+0000", 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319347+0000", 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4591999999999997e-05 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319431+0000", 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3893000000000005e-05 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13739 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.3254538530000008, 2024-09-16T13:17:31.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021506517999999999, 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13739 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:31.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298049+0000", 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298052+0000", 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.819e-06 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298058+0000", 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6960000000000002e-06 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298061+0000", 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7560000000000001e-06 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316465+0000", 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018403744 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316747+0000", 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028175200000000002 2024-09-16T13:17:31.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.317694+0000", 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000947115 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319304+0000", 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016102029999999999 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319446+0000", 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142008 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319490+0000", 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3690999999999999e-05 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319555+0000", 2024-09-16T13:17:31.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5191000000000007e-05 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13740 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.3254003700000005, 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023881870999999999, 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:31.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13740 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:31.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298103+0000", 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298106+0000", 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1559999999999999e-06 2024-09-16T13:17:31.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298120+0000", 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4474999999999999e-05 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298124+0000", 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1699999999999999e-06 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319635+0000", 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021510129999999999 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319762+0000", 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012746400000000001 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:31.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.320225+0000", 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046264800000000002 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321629+0000", 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00140383 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321682+0000", 2024-09-16T13:17:31.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3720000000000001e-05 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321757+0000", 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321984+0000", 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022684199999999999 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.769 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.770 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.770 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:31.770 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:31.770 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:31.771 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 51, 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "6ddcae89-6c18-4eb7-8712-1c53f3b07278", 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-16T13:16:23.176689+0000", 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-16T13:17:30.213574+0000", 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "last_up_change": "2024-09-16T13:16:33.765413+0000", 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "last_in_change": "2024-09-16T13:17:21.137991+0000", 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_num": 5806080, 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_set": [ 2024-09-16T13:17:31.772 INFO:teuthology.orchestra.run.smithi098.stdout: "nodeep-scrub", 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "noscrub", 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "pglog_hardlimit", 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "purged_snapdirs", 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_deletes", 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "sortbitwise" 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_version": 5, 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T13:17:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_snapshot": "", 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_max": 3, 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd": 8, 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "require_min_compat_client": "luminous", 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "min_compat_client": "luminous", 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "require_osd_release": "quincy", 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "pools": [ 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 1, 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": ".mgr", 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:33.497561+0000", 2024-09-16T13:17:31.774 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool", 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:17:31.775 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 1, 2024-09-16T13:17:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 1, 2024-09-16T13:17:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 1, 2024-09-16T13:17:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 1, 2024-09-16T13:17:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 1, 2024-09-16T13:17:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:17:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:17:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:17:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:17:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:17:31.776 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "17", 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 0, 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 0, 2024-09-16T13:17:31.777 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:17:31.778 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:17:31.779 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "options": { 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_max": 32, 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_min": 1 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:17:31.780 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": {} 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 2, 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": "rbd", 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:40.994538+0000", 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 8193, 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:17:31.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 8, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 8, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 8, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 8, 2024-09-16T13:17:31.782 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 8, 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "21", 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:17:31.783 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 3, 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 21, 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:17:31.784 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:17:31.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:17:31.786 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "options": {}, 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd": {} 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 3, 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": "pool_client.0", 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:50.759705+0000", 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 8193, 2024-09-16T13:17:31.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 32, 2024-09-16T13:17:31.788 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 32, 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 32, 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 32, 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 32, 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "50", 2024-09-16T13:17:31.789 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 16, 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 50, 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:17:31.790 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:17:31.791 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:17:31.792 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "options": {}, 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd": {} 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: "osds": [ 2024-09-16T13:17:31.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 0, 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "be8543b4-3d28-4b37-a4ce-d2a56a09aeaf", 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 0, 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 23, 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:31.794 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6808", 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6809", 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:31.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6810", 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6811", 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6816", 2024-09-16T13:17:31.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6817", 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6813", 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6814", 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6809/22271", 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6811/22271", 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6817/22271", 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6814/22271", 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.800 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 1, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "a7f95c7b-56d4-437b-a160-cabb8fd2b1e4", 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 43, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:31.801 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6812", 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6815", 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.802 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6818", 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6819", 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6822", 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6823", 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6820", 2024-09-16T13:17:31.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6821", 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6815/22272", 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6819/22272", 2024-09-16T13:17:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6823/22272", 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6821/22272", 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 2, 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c", 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:31.807 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 18, 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6800", 2024-09-16T13:17:31.809 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:31.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6801", 2024-09-16T13:17:31.809 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:31.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6802", 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6803", 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6806", 2024-09-16T13:17:31.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6807", 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6804", 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6805", 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6801/22273", 2024-09-16T13:17:31.813 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6803/22273", 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6807/22273", 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6805/22273", 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 3, 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "4898b5b5-b912-430c-adcb-c52d90226917", 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:31.814 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 46, 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6824", 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6825", 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.816 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6826", 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6827", 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.817 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6830", 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6831", 2024-09-16T13:17:31.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6828", 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6829", 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6825/22270", 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6827/22270", 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6831/22270", 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6829/22270", 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:31.820 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 4, 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "6c478be2-f606-41d8-967c-9a689f09a440", 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:31.821 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 46, 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6824", 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6825", 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6826", 2024-09-16T13:17:31.823 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6827", 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6830", 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6831", 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.825 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6828", 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6829", 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:31.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6825/22286", 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6827/22286", 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6831/22286", 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6829/22286", 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 5, 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "5001c828-b223-4d2c-b893-f9c56da73b83", 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 46, 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:31.828 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6808", 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6809", 2024-09-16T13:17:31.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6810", 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6811", 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6814", 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:31.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6815", 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6812", 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6813", 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6809/22285", 2024-09-16T13:17:31.833 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6811/22285", 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6815/22285", 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6813/22285", 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 6, 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "6f7a9125-674c-4be2-93e1-b6296cd00f8e", 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 43, 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6816", 2024-09-16T13:17:31.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:31.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6817", 2024-09-16T13:17:31.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:31.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.836 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6818", 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6819", 2024-09-16T13:17:31.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6822", 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6823", 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6820", 2024-09-16T13:17:31.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:31.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6821", 2024-09-16T13:17:31.840 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:31.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.840 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.840 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6817/22315", 2024-09-16T13:17:31.840 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6819/22315", 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6823/22315", 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6821/22315", 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 7, 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "bcdfe435-f24d-45a9-a629-7380ad2e575a", 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:31.841 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 43, 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6800", 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6801", 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6802", 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6803", 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6806", 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6807", 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6804", 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6805", 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6801/22284", 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6803/22284", 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6807/22284", 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6805/22284", 2024-09-16T13:17:31.847 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_xinfo": [ 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 0, 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:31.848 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 54531, 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.191311+0000", 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 1, 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:31.849 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.563551+0000", 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 2, 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.493457+0000", 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:31.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 3, 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.599815+0000", 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 4, 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.634352+0000", 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 5, 2024-09-16T13:17:31.852 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.262873+0000", 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 6, 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:31.853 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.508881+0000", 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 7, 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:31.854 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.348276+0000", 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_upmap": [ 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: "pgid": "3.1c", 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: "osds": [ 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: 1 2024-09-16T13:17:31.855 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_upmap_items": [], 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_temp": [], 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_temp": [], 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklist": {}, 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: "range_blocklist": {}, 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profiles": { 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: "default": { 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-failure-domain": "osd", 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: "k": "2", 2024-09-16T13:17:31.856 INFO:teuthology.orchestra.run.smithi098.stdout: "m": "1", 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: "plugin": "jerasure", 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: "technique": "reed_sol_van" 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps_queue": [], 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: "new_removed_snaps": [], 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: "new_purged_snaps": [], 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_node_flags": {}, 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: "device_class_flags": {}, 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": { 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode_enabled": false, 2024-09-16T13:17:31.857 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_bucket_count": 0, 2024-09-16T13:17:31.858 INFO:teuthology.orchestra.run.smithi098.stdout: "degraded_stretch_mode": 0, 2024-09-16T13:17:31.858 INFO:teuthology.orchestra.run.smithi098.stdout: "recovering_stretch_mode": 0, 2024-09-16T13:17:31.858 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode_bucket": 0 2024-09-16T13:17:31.858 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:31.858 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:31.858 INFO:teuthology.orchestra.run.smithi098.stdout:2319 clone 14 order 21 su 1048576 sc 10 2024-09-16T13:17:31.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:31.860 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:31.767+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:31.861 INFO:tasks.thrashosds.thrasher:j is {'epoch': 51, 'fsid': '6ddcae89-6c18-4eb7-8712-1c53f3b07278', 'created': '2024-09-16T13:16:23.176689+0000', 'modified': '2024-09-16T13:17:30.213574+0000', 'last_up_change': '2024-09-16T13:16:33.765413+0000', 'last_in_change': '2024-09-16T13:17:21.137991+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:16:33.497561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:16:40.994538+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-16T13:16:50.759705+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 16, 'snap_epoch': 50, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'be8543b4-3d28-4b37-a4ce-d2a56a09aeaf', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6808', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6809', 'nonce': 22271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6810', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6811', 'nonce': 22271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6816', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6817', 'nonce': 22271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6813', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6814', 'nonce': 22271}]}, 'public_addr': '172.21.15.98:6809/22271', 'cluster_addr': '172.21.15.98:6811/22271', 'heartbeat_back_addr': '172.21.15.98:6817/22271', 'heartbeat_front_addr': '172.21.15.98:6814/22271', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a7f95c7b-56d4-437b-a160-cabb8fd2b1e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 43, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6812', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6815', 'nonce': 22272}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6818', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6819', 'nonce': 22272}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6822', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6823', 'nonce': 22272}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6820', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6821', 'nonce': 22272}]}, 'public_addr': '172.21.15.98:6815/22272', 'cluster_addr': '172.21.15.98:6819/22272', 'heartbeat_back_addr': '172.21.15.98:6823/22272', 'heartbeat_front_addr': '172.21.15.98:6821/22272', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6800', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6801', 'nonce': 22273}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6802', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6803', 'nonce': 22273}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6806', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6807', 'nonce': 22273}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6804', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6805', 'nonce': 22273}]}, 'public_addr': '172.21.15.98:6801/22273', 'cluster_addr': '172.21.15.98:6803/22273', 'heartbeat_back_addr': '172.21.15.98:6807/22273', 'heartbeat_front_addr': '172.21.15.98:6805/22273', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4898b5b5-b912-430c-adcb-c52d90226917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6824', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6825', 'nonce': 22270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6826', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6827', 'nonce': 22270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6830', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6831', 'nonce': 22270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6828', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6829', 'nonce': 22270}]}, 'public_addr': '172.21.15.98:6825/22270', 'cluster_addr': '172.21.15.98:6827/22270', 'heartbeat_back_addr': '172.21.15.98:6831/22270', 'heartbeat_front_addr': '172.21.15.98:6829/22270', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6c478be2-f606-41d8-967c-9a689f09a440', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 22286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 22286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 22286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 22286}]}, 'public_addr': '172.21.15.151:6825/22286', 'cluster_addr': '172.21.15.151:6827/22286', 'heartbeat_back_addr': '172.21.15.151:6831/22286', 'heartbeat_front_addr': '172.21.15.151:6829/22286', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5001c828-b223-4d2c-b893-f9c56da73b83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 22285}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 22285}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 22285}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 22285}]}, 'public_addr': '172.21.15.151:6809/22285', 'cluster_addr': '172.21.15.151:6811/22285', 'heartbeat_back_addr': '172.21.15.151:6815/22285', 'heartbeat_front_addr': '172.21.15.151:6813/22285', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6f7a9125-674c-4be2-93e1-b6296cd00f8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 43, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 22315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 22315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 22315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 22315}]}, 'public_addr': '172.21.15.151:6817/22315', 'cluster_addr': '172.21.15.151:6819/22315', 'heartbeat_back_addr': '172.21.15.151:6823/22315', 'heartbeat_front_addr': '172.21.15.151:6821/22315', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bcdfe435-f24d-45a9-a629-7380ad2e575a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 43, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6800', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6801', 'nonce': 22284}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 22284}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 22284}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 22284}]}, 'public_addr': '172.21.15.151:6801/22284', 'cluster_addr': '172.21.15.151:6803/22284', 'heartbeat_back_addr': '172.21.15.151:6807/22284', 'heartbeat_front_addr': '172.21.15.151:6805/22284', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 54531, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.191311+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.563551+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.493457+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.599815+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.634352+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.262873+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.508881+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.348276+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.1c', 'osds': [6, 1]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T13:17:31.861 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-16T13:17:31.904 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:31.900+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:31.968 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:31.968 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:31.968 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:31.968 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:31.973 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:31.967+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:31.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:32.074 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:32.074 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:32.074 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:32.074 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:32.074 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:32.075 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:32.068+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:32.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:32.175 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:32.167+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:32.184 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2520 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155848842000001, 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012988767, 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.185 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2520 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024617+0000", 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024618+0000", 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0199999999999998e-07 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024620+0000", 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1150000000000001e-06 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024624+0000", 2024-09-16T13:17:32.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.676e-06 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024626+0000", 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6730000000000001e-06 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037443+0000", 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012816869 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037522+0000", 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9060000000000005e-05 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037606+0000", 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4072000000000004e-05 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2521 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2523136~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:32.189 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155837470000002, 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013552781999999999, 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2521 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024629+0000", 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024630+0000", 2024-09-16T13:17:32.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1629999999999999e-06 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024635+0000", 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5050000000000001e-06 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024637+0000", 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4040000000000002e-06 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037635+0000", 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012997754 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.037729+0000", 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4043999999999996e-05 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038182+0000", 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045262500000000001 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.193 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2522 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3473408~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155825647, 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013780114, 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2522 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.194 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8999999999999998e-07 2024-09-16T13:17:32.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024642+0000", 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3230000000000001e-06 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024646+0000", 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024648+0000", 2024-09-16T13:17:32.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4849999999999999e-06 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038251+0000", 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013602822000000001 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038335+0000", 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4215999999999995e-05 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038421+0000", 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.543e-05 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2523 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155812701999999, 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014015147, 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.198 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2523 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024656+0000", 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.131e-06 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024660+0000", 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6990000000000001e-06 2024-09-16T13:17:32.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024663+0000", 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0290000000000001e-06 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038456+0000", 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013793481 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038549+0000", 2024-09-16T13:17:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2608000000000005e-05 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038669+0000", 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011982800000000001 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2524 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155799128000002, 2024-09-16T13:17:32.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014381392, 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2524 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024668+0000", 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8400000000000003e-07 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024670+0000", 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.849e-06 2024-09-16T13:17:32.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024675+0000", 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.451e-06 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024678+0000", 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8739999999999999e-06 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038723+0000", 2024-09-16T13:17:32.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014044562999999999 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038811+0000", 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039049+0000", 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023761100000000001 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2525 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 557056~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155764780999998, 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014602618, 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2525 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:32.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024705+0000", 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4650000000000001e-06 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024711+0000", 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024716+0000", 2024-09-16T13:17:32.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1710000000000004e-06 2024-09-16T13:17:32.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039096+0000", 2024-09-16T13:17:32.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014380184000000001 2024-09-16T13:17:32.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039187+0000", 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1065999999999999e-05 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039304+0000", 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117333 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2526 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1507328~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.211 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155746612000002, 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014823846, 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2526 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:32.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024722+0000", 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024729+0000", 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024733+0000", 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039337+0000", 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014604026000000001 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039427+0000", 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0437000000000002e-05 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039544+0000", 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116234 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.215 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2527 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1605632~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155728848999999, 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015041116, 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2527 2024-09-16T13:17:32.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024738+0000", 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024746+0000", 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2130000000000008e-06 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024753+0000", 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024758+0000", 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7670000000000003e-06 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014820533 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039670+0000", 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1825999999999998e-05 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039779+0000", 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108424 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2528 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155703411000001, 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015245242000000001, 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.220 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.221 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.221 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.221 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.221 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2528 2024-09-16T13:17:32.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.221 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:32.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024765+0000", 2024-09-16T13:17:32.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1040000000000001e-06 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024772+0000", 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024776+0000", 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039810+0000", 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015034194000000001 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039897+0000", 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6864e-05 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040008+0000", 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110802 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.224 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2529 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2555904~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155684292, 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015471574, 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2529 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.225 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:17:32.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024785+0000", 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5969999999999999e-06 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024800+0000", 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4681999999999999e-05 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024806+0000", 2024-09-16T13:17:32.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040039+0000", 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015233007999999999 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040143+0000", 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010322799999999999 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040254+0000", 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011092899999999999 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2530 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155654011999999, 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015667606000000001, 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.229 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2530 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024813+0000", 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024815+0000", 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5160000000000001e-06 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024827+0000", 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1915e-05 2024-09-16T13:17:32.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024834+0000", 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5180000000000001e-06 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040286+0000", 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015452565999999999 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040375+0000", 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040480+0000", 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105388 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2531 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3538944~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155626797, 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015867612999999999, 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2531 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024843+0000", 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6960000000000001e-06 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024856+0000", 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-05 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024863+0000", 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6069999999999996e-06 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040513+0000", 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650662999999999 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040599+0000", 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5439999999999995e-05 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040707+0000", 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108461 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.237 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2532 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155596291999998, 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061912000000001, 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2532 2024-09-16T13:17:32.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024871+0000", 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024874+0000", 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024881+0000", 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0570000000000004e-06 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024886+0000", 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040742+0000", 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015856808 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040824+0000", 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:17:32.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040932+0000", 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108378 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2533 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155575766999998, 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016272282999999998, 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.242 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2533 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024893+0000", 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419999999999999e-06 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024900+0000", 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024903+0000", 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040965+0000", 2024-09-16T13:17:32.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061565999999999 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041052+0000", 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7213000000000007e-05 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041163+0000", 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011066899999999999 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2534 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3604480~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155557236, 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016482746999999999, 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2534 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024910+0000", 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1299999999999999e-07 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024912+0000", 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024920+0000", 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024924+0000", 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.036e-06 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041197+0000", 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016273242 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041282+0000", 2024-09-16T13:17:32.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5013999999999996e-05 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041392+0000", 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109567 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2535 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155536573999999, 2024-09-16T13:17:32.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016691902000000002, 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2535 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7599999999999997e-07 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024932+0000", 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-16T13:17:32.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024949+0000", 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7053000000000001e-05 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024961+0000", 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041425+0000", 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016463851000000002 2024-09-16T13:17:32.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041515+0000", 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9549999999999995e-05 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041622+0000", 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107132 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2536 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1638400~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155498768000001, 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016880869999999999, 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2536 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024971+0000", 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.615e-06 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024988+0000", 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6912999999999999e-05 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024993+0000", 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-09-16T13:17:32.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041653+0000", 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016660245000000001 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041743+0000", 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0099999999999995e-05 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041848+0000", 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105404 2024-09-16T13:17:32.259 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2537 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1671168~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155468015, 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017075346000000002, 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.260 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2537 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024999+0000", 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025002+0000", 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025009+0000", 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-09-16T13:17:32.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025013+0000", 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1840000000000001e-06 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041886+0000", 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016873249999999999 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041971+0000", 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4487000000000006e-05 2024-09-16T13:17:32.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042074+0000", 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102847 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2538 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2621440~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155447844000001, 2024-09-16T13:17:32.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017314057000000001, 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2538 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025019+0000", 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025021+0000", 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025033+0000", 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1985e-05 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025043+0000", 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3810000000000005e-06 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042107+0000", 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017064471000000001 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042203+0000", 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5810999999999995e-05 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042333+0000", 2024-09-16T13:17:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012956099999999999 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2539 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.155420115999998, 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017500180000000001, 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2539 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025047+0000", 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025053+0000", 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025058+0000", 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025061+0000", 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.072e-06 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042365+0000", 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017304885999999998 2024-09-16T13:17:32.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042458+0000", 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2991000000000005e-05 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042546+0000", 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7904000000000004e-05 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.273 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.274 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:32.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:17:32.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:32.318 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:32.311+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:32.377 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:32.372+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:32.420 INFO:teuthology.orchestra.run.smithi098.stdout:flattening image image_client.0-clone13 2024-09-16T13:17:32.433 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:32.434 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:32.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:32.478 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:32.471+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:32.542 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:32.542 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:32.542 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:32.542 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:32.542 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:32.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:32.579 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:32.572+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:32.655 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:32.655 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:32.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:32.655 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:32.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.655 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11487 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 430080~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.655 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:32.655 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.162664802, 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014286111000000001, 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11487 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:32.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486860+0000", 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.04e-06 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486866+0000", 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0629999999999996e-06 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486881+0000", 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5023e-05 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486889+0000", 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8149999999999995e-06 2024-09-16T13:17:32.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.500906+0000", 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014016416 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501008+0000", 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102243 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501146+0000", 2024-09-16T13:17:32.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013751099999999999 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11535 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.161981829, 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014337727999999999, 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.660 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11535 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487543+0000", 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023e-06 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487551+0000", 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0309999999999998e-06 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487566+0000", 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5000999999999999e-05 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487578+0000", 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1066e-05 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501655+0000", 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014077151 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501747+0000", 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2670999999999994e-05 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501880+0000", 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013278500000000001 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11541 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 413696~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.664 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.161937279, 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014490381, 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11541 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:32.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487588+0000", 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487592+0000", 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4920000000000004e-06 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487607+0000", 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4926000000000001e-05 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487616+0000", 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6829999999999997e-06 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501906+0000", 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0142898 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501959+0000", 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2985000000000002e-05 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502077+0000", 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118495 2024-09-16T13:17:32.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11542 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 393216~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.161898753999999, 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014676434, 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11542 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487626+0000", 2024-09-16T13:17:32.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.034e-06 2024-09-16T13:17:32.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487631+0000", 2024-09-16T13:17:32.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7659999999999998e-06 2024-09-16T13:17:32.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487646+0000", 2024-09-16T13:17:32.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4786e-05 2024-09-16T13:17:32.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487655+0000", 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6489999999999997e-06 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502110+0000", 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014455749 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502209+0000", 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8608000000000002e-05 2024-09-16T13:17:32.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502302+0000", 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2842000000000003e-05 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11544 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 466944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.161857792999999, 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014846796000000001, 2024-09-16T13:17:32.674 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11544 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487667+0000", 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.569999999999999e-07 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487672+0000", 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2459999999999997e-06 2024-09-16T13:17:32.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487686+0000", 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4847e-05 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487695+0000", 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9109999999999999e-06 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502337+0000", 2024-09-16T13:17:32.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014641861000000001 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502440+0000", 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010282599999999999 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502513+0000", 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3147999999999998e-05 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11547 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.161818666, 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015113645, 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11547 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487707+0000", 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.015e-06 2024-09-16T13:17:32.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487711+0000", 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8289999999999998e-06 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487727+0000", 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5497000000000001e-05 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487736+0000", 2024-09-16T13:17:32.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1409999999999994e-06 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502557+0000", 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014821212 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502675+0000", 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117741 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502819+0000", 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014421 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11549 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 413696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.161778244000001, 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016244525999999999, 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.683 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11549 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487747+0000", 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.094e-06 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487752+0000", 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.758e-06 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487770+0000", 2024-09-16T13:17:32.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8502999999999999e-05 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487779+0000", 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1579999999999994e-06 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502868+0000", 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015088811000000001 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502916+0000", 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8109000000000003e-05 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503990+0000", 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010740929999999999 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11578 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 409600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.161727802, 2024-09-16T13:17:32.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015410814, 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11578 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487798+0000", 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.646e-06 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487804+0000", 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487821+0000", 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7677999999999999e-05 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487830+0000", 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9090000000000006e-06 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503033+0000", 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015202337999999999 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503123+0000", 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0414000000000005e-05 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503207+0000", 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4172000000000007e-05 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11580 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.691 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.161684166000001, 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015544980999999999, 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11580 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487841+0000", 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487846+0000", 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3919999999999996e-06 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487861+0000", 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4946000000000001e-05 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487869+0000", 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8750000000000006e-06 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503240+0000", 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015370511999999999 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503334+0000", 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4265000000000006e-05 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503385+0000", 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0773999999999999e-05 2024-09-16T13:17:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11617 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 430080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.160364467999999, 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376203000000001, 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11617 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489162+0000", 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3089999999999998e-06 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489171+0000", 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3430000000000002e-06 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489198+0000", 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6449999999999999e-05 2024-09-16T13:17:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489216+0000", 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8057999999999999e-05 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503402+0000", 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014186196999999999 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503454+0000", 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1989e-05 2024-09-16T13:17:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503536+0000", 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1856999999999999e-05 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11624 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 503808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.16016138, 2024-09-16T13:17:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376676, 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11624 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489365+0000", 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.795e-06 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489372+0000", 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4030000000000001e-06 2024-09-16T13:17:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489392+0000", 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0129e-05 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489404+0000", 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1555e-05 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503558+0000", 2024-09-16T13:17:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014154771999999999 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503644+0000", 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5691000000000003e-05 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503740+0000", 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5330999999999999e-05 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11630 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.160108042999999, 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014893777, 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11630 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489417+0000", 2024-09-16T13:17:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6800000000000009e-07 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489435+0000", 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7818000000000001e-05 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489450+0000", 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5072000000000001e-05 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489459+0000", 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3009999999999995e-06 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503787+0000", 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01432719 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503889+0000", 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102604 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504310+0000", 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042082399999999998 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11632 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.160054929999999, 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01524066, 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11632 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489470+0000", 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.78e-07 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489475+0000", 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9400000000000001e-06 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489488+0000", 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3176999999999999e-05 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489497+0000", 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4919999999999993e-06 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504364+0000", 2024-09-16T13:17:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014866846 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504485+0000", 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012142999999999999 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504710+0000", 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000224897 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11639 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.160013159, 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015375941000000001, 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11639 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489512+0000", 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:17:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489517+0000", 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.938e-06 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489532+0000", 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.47e-05 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489540+0000", 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2949999999999994e-06 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504765+0000", 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015225449 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504832+0000", 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6929999999999998e-05 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504887+0000", 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4681e-05 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11641 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.159966648999999, 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015522312999999999, 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11641 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489559+0000", 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.066e-06 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489564+0000", 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9370000000000003e-06 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489579+0000", 2024-09-16T13:17:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5466000000000001e-05 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489588+0000", 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504925+0000", 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015337353 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505025+0000", 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9770000000000002e-05 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505080+0000", 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4792e-05 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11667 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 16384~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.159605880000001, 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015353510000000001, 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11667 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489920+0000", 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.84e-06 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489928+0000", 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6969999999999993e-06 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489953+0000", 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5154999999999999e-05 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489965+0000", 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2292e-05 2024-09-16T13:17:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505115+0000", 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015149793999999999 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505184+0000", 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8858999999999999e-05 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505272+0000", 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7873000000000004e-05 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11672 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 36864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.159545475, 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015615053, 2024-09-16T13:17:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11672 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489980+0000", 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489985+0000", 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490000+0000", 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490010+0000", 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.183e-06 2024-09-16T13:17:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505335+0000", 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015325167000000001 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505471+0000", 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013581699999999999 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505594+0000", 2024-09-16T13:17:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123168 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11676 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.159504731, 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015779395000000002, 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11676 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490021+0000", 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.575e-06 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490026+0000", 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5170000000000001e-06 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490041+0000", 2024-09-16T13:17:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4955e-05 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490049+0000", 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5520000000000005e-06 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505647+0000", 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015597489000000001 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505743+0000", 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6908000000000001e-05 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505799+0000", 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5399000000000003e-05 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11678 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.159465458, 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015992643000000001, 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11678 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490060+0000", 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3400000000000001e-06 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490065+0000", 2024-09-16T13:17:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4830000000000001e-06 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490079+0000", 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4854999999999999e-05 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490088+0000", 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1119999999999996e-06 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505849+0000", 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015761718000000001 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505966+0000", 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011645399999999999 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506051+0000", 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5680999999999995e-05 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11707 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 53248~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.159246329, 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016106869999999999, 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11707 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490280+0000", 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7659999999999999e-06 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490287+0000", 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.503e-06 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490307+0000", 2024-09-16T13:17:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9989000000000001e-05 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490318+0000", 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0967e-05 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506171+0000", 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015853367 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506265+0000", 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3190999999999996e-05 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506385+0000", 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120087 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:32.745 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:32.746 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #12, image image_client.0-clone12 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent13 2024-09-16T13:17:32.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:32.748 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:32.740+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:32.786 INFO:teuthology.orchestra.run.smithi098.stdout:2320 write 0x9b94a65 thru 0x9b9bee8 (0x7484 bytes) 2024-09-16T13:17:32.801 INFO:teuthology.orchestra.run.smithi098.stdout:2321 read 0xa29c2f thru 0xa2ada8 (0x117a bytes) 2024-09-16T13:17:32.801 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:32.801 INFO:teuthology.orchestra.run.smithi098.stdout:2324 punch from 0x6fb8ec0 to 0x6fc65ef, (0xd72f bytes) 2024-09-16T13:17:32.802 INFO:teuthology.orchestra.run.smithi098.stdout:2325 read 0xaa8495f thru 0xaa89e13 (0x54b5 bytes) 2024-09-16T13:17:32.812 INFO:teuthology.orchestra.run.smithi098.stdout:2326 trunc from 0xdb4403e to 0x86b62f6 2024-09-16T13:17:32.847 INFO:teuthology.orchestra.run.smithi098.stdout:2327 punch from 0x55d77a0 to 0x55e0a45, (0x92a5 bytes) 2024-09-16T13:17:32.847 INFO:teuthology.orchestra.run.smithi098.stdout:2328 write 0x739215 thru 0x73b613 (0x23ff bytes) 2024-09-16T13:17:32.847 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:32.840+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:32.858 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:32.858 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:32.858 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:32.858 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:32.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:32.872 INFO:teuthology.orchestra.run.smithi098.stdout:2332 write 0xc351a1b thru 0xc35c290 (0xa876 bytes) 2024-09-16T13:17:32.879 INFO:teuthology.orchestra.run.smithi098.stdout:2333 read 0x42d8800 thru 0x42e510c (0xc90d bytes) 2024-09-16T13:17:32.880 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:32.880 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:32.880 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:32.880 INFO:teuthology.orchestra.run.smithi098.stdout:2338 punch from 0x1b80731 to 0x1b8f443, (0xed12 bytes) 2024-09-16T13:17:32.880 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:32.880 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:32.880 INFO:teuthology.orchestra.run.smithi098.stdout:2342 read 0x284a29d thru 0x284d6a8 (0x340c bytes) 2024-09-16T13:17:32.880 INFO:teuthology.orchestra.run.smithi098.stdout:2343 write 0x7290d54 thru 0x7294ea8 (0x4155 bytes) 2024-09-16T13:17:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:2344 punch from 0xb32944f to 0xb329cb1, (0x862 bytes) 2024-09-16T13:17:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:2345 punch from 0x4426937 to 0x44358e9, (0xefb2 bytes) 2024-09-16T13:17:32.888 INFO:teuthology.orchestra.run.smithi098.stdout:2346 write 0x261f671 thru 0x26229ed (0x337d bytes) 2024-09-16T13:17:32.895 INFO:teuthology.orchestra.run.smithi098.stdout:2347 read 0x9a88303 thru 0x9a8f1de (0x6edc bytes) 2024-09-16T13:17:32.895 INFO:teuthology.orchestra.run.smithi098.stdout:2349 read 0x41a5416 thru 0x41a69c1 (0x15ac bytes) 2024-09-16T13:17:32.895 INFO:teuthology.orchestra.run.smithi098.stdout:2350 writesame 0x56fd189 thru 0x5706dcf data_size 0x1(0x9c47 bytes) 2024-09-16T13:17:32.930 INFO:teuthology.orchestra.run.smithi098.stdout:2351 read 0x13ae079 thru 0x13b1023 (0x2fab bytes) 2024-09-16T13:17:32.942 INFO:teuthology.orchestra.run.smithi098.stdout:2352 write 0xc3661a0 thru 0xc36cb51 (0x69b2 bytes) 2024-09-16T13:17:32.945 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:32.946 INFO:teuthology.orchestra.run.smithi098.stdout:2354 read 0x20e5cef thru 0x20f3d71 (0xe083 bytes) 2024-09-16T13:17:32.946 INFO:teuthology.orchestra.run.smithi098.stdout:2355 read 0xa7e19b3 thru 0xa7ee1e5 (0xc833 bytes) 2024-09-16T13:17:32.947 INFO:teuthology.orchestra.run.smithi098.stdout:2356 write 0x7016f2b thru 0x70220f6 (0xb1cc bytes) 2024-09-16T13:17:32.947 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:32.939+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:32.954 INFO:teuthology.orchestra.run.smithi098.stdout:2357 writesame 0x661adf3 thru 0x6623e87 data_size 0x1(0x9095 bytes) 2024-09-16T13:17:32.959 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:32.959 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:32.959 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:32.959 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:32.959 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:32.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:32.987 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:32.987 INFO:teuthology.orchestra.run.smithi098.stdout:2359 write 0x68c04f0 thru 0x68c37c3 (0x32d4 bytes) 2024-09-16T13:17:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:2360 punch from 0x1294c1c to 0x12a1f78, (0xd35c bytes) 2024-09-16T13:17:32.991 INFO:teuthology.orchestra.run.smithi098.stdout:2361 punch from 0xefe8a7 to 0xf0326a, (0x49c3 bytes) 2024-09-16T13:17:32.995 INFO:teuthology.orchestra.run.smithi098.stdout:2362 trunc from 0xc36cb52 to 0xa47b5f5 2024-09-16T13:17:33.021 INFO:teuthology.orchestra.run.smithi098.stdout:2363 read 0x13b3ad7 thru 0x13bb171 (0x769b bytes) 2024-09-16T13:17:33.021 INFO:teuthology.orchestra.run.smithi098.stdout:2364 write 0xb681fb thru 0xb713f6 (0x91fc bytes) 2024-09-16T13:17:33.048 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:33.039+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:33.050 INFO:teuthology.orchestra.run.smithi098.stdout:2365 write 0xc4434ee thru 0xc446eb8 (0x39cb bytes) 2024-09-16T13:17:33.057 INFO:teuthology.orchestra.run.smithi098.stdout:2366 write 0x1aa9733 thru 0x1aab721 (0x1fef bytes) 2024-09-16T13:17:33.064 INFO:teuthology.orchestra.run.smithi098.stdout:2368 punch from 0x1f3aaf3 to 0x1f415d5, (0x6ae2 bytes) 2024-09-16T13:17:33.073 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:33.073 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 58.041346685999997, 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:17:33.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:17:33.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:17:33.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 58.041199280000001, 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.080 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:17:33.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:17:33.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:17:33.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 58.041171937000001, 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:17:33.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:17:33.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:17:33.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:33.093 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 57.482496675, 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:17:33.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:17:33.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 57.482409083, 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:17:33.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:17:33.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:17:33.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:33.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:33.105 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 57.482239161000003, 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:17:33.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:17:33.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2259 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3014656~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330910585000002, 2024-09-16T13:17:33.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016650893999999999, 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2259 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735831+0000", 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9530000000000005e-06 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735836+0000", 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7450000000000003e-06 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735839+0000", 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.915e-06 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752296+0000", 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016457241000000001 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752360+0000", 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4108000000000004e-05 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752473+0000", 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112305 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2260 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330890664000002, 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016799073000000001, 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.116 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2260 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:33.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735843+0000", 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3480000000000001e-06 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735847+0000", 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5990000000000002e-06 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735849+0000", 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5859999999999999e-06 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752509+0000", 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016659377 2024-09-16T13:17:33.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752559+0000", 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9774000000000002e-05 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752641+0000", 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2057000000000004e-05 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2261 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330878474999999, 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016960698, 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2261 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.03e-07 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735855+0000", 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735860+0000", 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1110000000000001e-06 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735862+0000", 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:33.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752695+0000", 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01683343 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752744+0000", 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8414000000000001e-05 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752814+0000", 2024-09-16T13:17:33.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0623000000000005e-05 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2262 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1048576~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330867861000002, 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017224107999999998, 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2262 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735865+0000", 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.91e-07 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735866+0000", 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1480000000000001e-06 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735870+0000", 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:17:33.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735877+0000", 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4370000000000001e-06 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752871+0000", 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016994061000000001 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752962+0000", 2024-09-16T13:17:33.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0470000000000006e-05 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753089+0000", 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126642 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2263 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330852402000001, 2024-09-16T13:17:33.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017392088999999999, 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2263 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-07 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735881+0000", 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735885+0000", 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4180000000000001e-06 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735887+0000", 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-06 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753116+0000", 2024-09-16T13:17:33.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017228963 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753184+0000", 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7374999999999998e-05 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753272+0000", 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8323000000000001e-05 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.133 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2264 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2064384~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330842551, 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017630541999999999, 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2264 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.134 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:33.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735891+0000", 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735895+0000", 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735897+0000", 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4669999999999998e-06 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753316+0000", 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017418685 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753403+0000", 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7590000000000007e-05 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753520+0000", 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116863 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2265 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1081344~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330832668999999, 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017739263000000002, 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2265 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735901+0000", 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735904+0000", 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.304e-06 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735907+0000", 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3599999999999999e-06 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753534+0000", 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017627446000000001 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753576+0000", 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1474999999999998e-05 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753639+0000", 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3188999999999994e-05 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2266 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2097152~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:33.142 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330823435999999, 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017970893000000002, 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2266 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735914+0000", 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1699999999999996e-06 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735918+0000", 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6859999999999999e-06 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735920+0000", 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4870000000000001e-06 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753685+0000", 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017764582000000001 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753771+0000", 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6259000000000002e-05 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753880+0000", 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010844300000000001 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2267 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.146 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330809535, 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018110804000000001, 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2267 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735924+0000", 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735928+0000", 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.416e-06 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735930+0000", 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:17:33.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753898+0000", 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017967852999999999 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753955+0000", 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6662000000000001e-05 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754034+0000", 2024-09-16T13:17:33.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8904000000000002e-05 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2268 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3080192~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330800050000001, 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018252866, 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2268 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735934+0000", 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0950000000000001e-06 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735938+0000", 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:17:33.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735941+0000", 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8750000000000002e-06 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754071+0000", 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018129941 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754120+0000", 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8359000000000002e-05 2024-09-16T13:17:33.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754185+0000", 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5358999999999995e-05 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2269 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330788671000001, 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018533516, 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.155 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2269 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735945+0000", 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1149999999999999e-06 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735948+0000", 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5250000000000001e-06 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735951+0000", 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9909999999999998e-06 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754270+0000", 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018318081999999999 2024-09-16T13:17:33.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754325+0000", 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5648e-05 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754477+0000", 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015188199999999999 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2270 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1114112~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330778202000001, 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019103282999999999, 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2270 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735955+0000", 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735956+0000", 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.096e-06 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735959+0000", 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5140000000000002e-06 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735963+0000", 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.053e-06 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754822+0000", 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018858883 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754941+0000", 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118789 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755057+0000", 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116272 2024-09-16T13:17:33.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2271 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1146880~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.330756390000001, 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019332309999999998, 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.164 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2271 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3500000000000002e-07 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735982+0000", 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735987+0000", 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1930000000000003e-06 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735993+0000", 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4659999999999996e-06 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755081+0000", 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019088187 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755194+0000", 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011223299999999999 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755308+0000", 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114663 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2272 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 163840~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.33073456, 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019476291, 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:33.168 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2272 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:33.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736003+0000", 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2279999999999998e-06 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736007+0000", 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.506e-06 2024-09-16T13:17:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736011+0000", 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9280000000000003e-06 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755336+0000", 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019325189999999999 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755393+0000", 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7293999999999997e-05 2024-09-16T13:17:33.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:33.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:33.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:33.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755474+0000", 2024-09-16T13:17:33.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0635000000000003e-05 2024-09-16T13:17:33.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.172 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:33.172 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:33.172 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:33.174 INFO:teuthology.orchestra.run.smithi098.stdout:2369 write 0xd95d4fd thru 0xd95f501 (0x2005 bytes) 2024-09-16T13:17:33.174 INFO:teuthology.orchestra.run.smithi098.stdout:2370 read 0xcbc433f thru 0xcbd3892 (0xf554 bytes) 2024-09-16T13:17:33.174 INFO:teuthology.orchestra.run.smithi098.stdout:2371 read 0x63f6283 thru 0x640346b (0xd1e9 bytes) 2024-09-16T13:17:33.174 INFO:teuthology.orchestra.run.smithi098.stdout:2372 writesame 0x4da7e6b thru 0x4dabc4c data_size 0x2(0x3de2 bytes) 2024-09-16T13:17:33.174 INFO:teuthology.orchestra.run.smithi098.stdout:2373 clone 15 order 21 su 524288 sc 2 2024-09-16T13:17:33.175 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:33.168+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:33.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:33.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:17:33.274 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:33.267+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:33.294 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:33.294 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:33.295 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:33.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:33.364 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone14 from 0xd95f502 (overlap 0x86b62f6) to 0x5e4c545 2024-09-16T13:17:33.375 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:33.367+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:33.385 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:33.404 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:33.410 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:33.410 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:33.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:33.421 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:33.447 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:33.476 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:33.472+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:33.480 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:33.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:33.517 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:33.518 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:33.523 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:33.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:33.546 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:33.549 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:33.600 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:33.596+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:33.638 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:33.638 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:33.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:33.667 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #13, image image_client.0-clone13 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent14 2024-09-16T13:17:33.679 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:33.672+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:33.750 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:33.750 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:33.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:33.779 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:33.776+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:33.868 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:33.868 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:33.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:33.881 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:33.875+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:33.982 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:33.976+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:34.022 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:34.022 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:34.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:34.083 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:34.075+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:34.114 INFO:teuthology.orchestra.run.smithi098.stdout:2374 punch from 0x9cdb04b to 0x9ce195a, (0x690f bytes) 2024-09-16T13:17:34.123 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.123 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.129 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:34.129 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:34.135 INFO:teuthology.orchestra.run.smithi098.stdout:2379 write 0xdf7d78e thru 0xdf7e9dc (0x124f bytes) 2024-09-16T13:17:34.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:34.141 INFO:teuthology.orchestra.run.smithi098.stdout:2380 write 0x87893c thru 0x87a8e6 (0x1fab bytes) 2024-09-16T13:17:34.179 INFO:teuthology.orchestra.run.smithi098.stdout:2381 trunc from 0xdf7e9dd to 0x25b3d8a 2024-09-16T13:17:34.184 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:34.175+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:34.224 INFO:teuthology.orchestra.run.smithi098.stdout:2382 write 0x3eaf7ae thru 0x3eb925a (0x9aad bytes) 2024-09-16T13:17:34.230 INFO:teuthology.orchestra.run.smithi098.stdout:2383 read 0xbaf648 thru 0xbbb8bf (0xc278 bytes) 2024-09-16T13:17:34.231 INFO:teuthology.orchestra.run.smithi098.stdout:2385 punch from 0x37b9cfb to 0x37be30f, (0x4614 bytes) 2024-09-16T13:17:34.231 INFO:teuthology.orchestra.run.smithi098.stdout:2387 write 0x1b640ae thru 0x1b69a79 (0x59cc bytes) 2024-09-16T13:17:34.239 INFO:teuthology.orchestra.run.smithi098.stdout:2388 punch from 0x25ca7af to 0x25d929b, (0xeaec bytes) 2024-09-16T13:17:34.240 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.240 INFO:teuthology.orchestra.run.smithi098.stdout:2390 punch from 0x3016cc to 0x3031f2, (0x1b26 bytes) 2024-09-16T13:17:34.240 INFO:teuthology.orchestra.run.smithi098.stdout:2391 trunc from 0x3eb925b to 0x40d774a 2024-09-16T13:17:34.240 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:34.241 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:34.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:34.252 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.252 INFO:teuthology.orchestra.run.smithi098.stdout:2394 punch from 0x37bfa1f to 0x37c2b9e, (0x317f bytes) 2024-09-16T13:17:34.275 INFO:teuthology.orchestra.run.smithi098.stdout:2395 writesame 0x89d0f6e thru 0x89df7de data_size 0x5(0xe871 bytes) 2024-09-16T13:17:34.275 INFO:teuthology.orchestra.run.smithi098.stdout:2398 writesame 0x6721c6a thru 0x672f0c0 data_size 0x1(0xd457 bytes) 2024-09-16T13:17:34.284 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:34.280+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:34.329 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.329 INFO:teuthology.orchestra.run.smithi098.stdout:2400 write 0xe6c6d6 thru 0xe6eb12 (0x243d bytes) 2024-09-16T13:17:34.348 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:34.348 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:34.352 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.353 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:34.368 INFO:teuthology.orchestra.run.smithi098.stdout:2404 write 0xa0692a8 thru 0xa06e22c (0x4f85 bytes) 2024-09-16T13:17:34.374 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.375 INFO:teuthology.orchestra.run.smithi098.stdout:2406 write 0x7bdb2e1 thru 0x7be902c (0xdd4c bytes) 2024-09-16T13:17:34.382 INFO:teuthology.orchestra.run.smithi098.stdout:2407 read 0x71d2830 thru 0x71d3507 (0xcd8 bytes) 2024-09-16T13:17:34.385 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:34.380+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:34.399 INFO:teuthology.orchestra.run.smithi098.stdout:2408 write 0xbcb87c7 thru 0xbcc1ed4 (0x970e bytes) 2024-09-16T13:17:34.405 INFO:teuthology.orchestra.run.smithi098.stdout:2409 read 0x2cc6224 thru 0x2ccffae (0x9d8b bytes) 2024-09-16T13:17:34.406 INFO:teuthology.orchestra.run.smithi098.stdout:2410 write 0xaf95ee3 thru 0xaf9dc4f (0x7d6d bytes) 2024-09-16T13:17:34.413 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.414 INFO:teuthology.orchestra.run.smithi098.stdout:2415 read 0x1d5646b thru 0x1d5b7ca (0x5360 bytes) 2024-09-16T13:17:34.414 INFO:teuthology.orchestra.run.smithi098.stdout:2416 read 0x22d9ff0 thru 0x22e4a33 (0xaa44 bytes) 2024-09-16T13:17:34.415 INFO:teuthology.orchestra.run.smithi098.stdout:2417 write 0x12dfa81 thru 0x12ede29 (0xe3a9 bytes) 2024-09-16T13:17:34.435 INFO:teuthology.orchestra.run.smithi098.stdout:2418 trunc from 0xbcc1ed5 to 0x5815c25 2024-09-16T13:17:34.472 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:34.472 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:34.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:34.482 INFO:teuthology.orchestra.run.smithi098.stdout:2420 write 0x97b1c86 thru 0x97b5786 (0x3b01 bytes) 2024-09-16T13:17:34.486 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:34.479+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:34.487 INFO:teuthology.orchestra.run.smithi098.stdout:2422 writesame 0x4d4f642 thru 0x4d56758 data_size 0x11(0x7117 bytes) 2024-09-16T13:17:34.497 INFO:teuthology.orchestra.run.smithi098.stdout:2426 read 0x2398b7b thru 0x23a356f (0xa9f5 bytes) 2024-09-16T13:17:34.497 INFO:teuthology.orchestra.run.smithi098.stdout:2427 read 0x4c6b49f thru 0x4c6de19 (0x297b bytes) 2024-09-16T13:17:34.498 INFO:teuthology.orchestra.run.smithi098.stdout:2428 writesame 0x203fcde thru 0x204f283 data_size 0x8d(0xf5a6 bytes) 2024-09-16T13:17:34.510 INFO:teuthology.orchestra.run.smithi098.stdout:2429 punch from 0x7597a69 to 0x759f4e1, (0x7a78 bytes) 2024-09-16T13:17:34.510 INFO:teuthology.orchestra.run.smithi098.stdout:2430 read 0x6e17658 thru 0x6e18ffb (0x19a4 bytes) 2024-09-16T13:17:34.510 INFO:teuthology.orchestra.run.smithi098.stdout:2431 write 0x5587e5 thru 0x559d34 (0x1550 bytes) 2024-09-16T13:17:34.525 INFO:teuthology.orchestra.run.smithi098.stdout:2432 punch from 0x563d517 to 0x564246c, (0x4f55 bytes) 2024-09-16T13:17:34.526 INFO:teuthology.orchestra.run.smithi098.stdout:2433 write 0x87c08ee thru 0x87c730a (0x6a1d bytes) 2024-09-16T13:17:34.532 INFO:teuthology.orchestra.run.smithi098.stdout:2435 read 0x8a4d3bd thru 0x8a532a8 (0x5eec bytes) 2024-09-16T13:17:34.532 INFO:teuthology.orchestra.run.smithi098.stdout:2436 write 0x26adc5e thru 0x26b47eb (0x6b8e bytes) 2024-09-16T13:17:34.535 INFO:teuthology.orchestra.run.smithi098.stdout:2437 punch from 0x1a1fb77 to 0x1a20d59, (0x11e2 bytes) 2024-09-16T13:17:34.535 INFO:teuthology.orchestra.run.smithi098.stdout:2438 writesame 0x5a7745c thru 0x5a83bcc data_size 0x5d(0xc771 bytes) 2024-09-16T13:17:34.559 INFO:teuthology.orchestra.run.smithi098.stdout:2439 write 0xac50587 thru 0xac5a292 (0x9d0c bytes) 2024-09-16T13:17:34.566 INFO:teuthology.orchestra.run.smithi098.stdout:2440 punch from 0x41c7792 to 0x41d30fe, (0xb96c bytes) 2024-09-16T13:17:34.566 INFO:teuthology.orchestra.run.smithi098.stdout:2441 punch from 0x956bf73 to 0x9573bff, (0x7c8c bytes) 2024-09-16T13:17:34.566 INFO:teuthology.orchestra.run.smithi098.stdout:2442 read 0x540bf04 thru 0x540d215 (0x1312 bytes) 2024-09-16T13:17:34.566 INFO:teuthology.orchestra.run.smithi098.stdout:2443 writesame 0x6cc503f thru 0x6cc6286 data_size 0x6(0x1248 bytes) 2024-09-16T13:17:34.573 INFO:teuthology.orchestra.run.smithi098.stdout:2444 read 0x64c8545 thru 0x64cd626 (0x50e2 bytes) 2024-09-16T13:17:34.573 INFO:teuthology.orchestra.run.smithi098.stdout:2445 read 0x52eb806 thru 0x52efd76 (0x4571 bytes) 2024-09-16T13:17:34.574 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.574 INFO:teuthology.orchestra.run.smithi098.stdout:2447 read 0x85ca04d thru 0x85d4bff (0xabb3 bytes) 2024-09-16T13:17:34.575 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.575 INFO:teuthology.orchestra.run.smithi098.stdout:2450 read 0x3c2f3f3 thru 0x3c3350b (0x4119 bytes) 2024-09-16T13:17:34.575 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.575 INFO:teuthology.orchestra.run.smithi098.stdout:2452 write 0x7d6edee thru 0x7d7b0e8 (0xc2fb bytes) 2024-09-16T13:17:34.581 INFO:teuthology.orchestra.run.smithi098.stdout:2453 punch from 0x7345318 to 0x734be12, (0x6afa bytes) 2024-09-16T13:17:34.581 INFO:teuthology.orchestra.run.smithi098.stdout:2454 write 0x535caaa thru 0x536a6b4 (0xdc0b bytes) 2024-09-16T13:17:34.587 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:34.584+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:34.588 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.588 INFO:teuthology.orchestra.run.smithi098.stdout:2461 write 0x2b3b048 thru 0x2b3f9c1 (0x497a bytes) 2024-09-16T13:17:34.594 INFO:teuthology.orchestra.run.smithi098.stdout:2463 punch from 0x56700b7 to 0x567d333, (0xd27c bytes) 2024-09-16T13:17:34.594 INFO:teuthology.orchestra.run.smithi098.stdout:2465 punch from 0x9023fe4 to 0x902bfee, (0x800a bytes) 2024-09-16T13:17:34.594 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.594 INFO:teuthology.orchestra.run.smithi098.stdout:2467 writesame 0x447f081 thru 0x448d5a8 data_size 0x4(0xe528 bytes) 2024-09-16T13:17:34.600 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:34.600 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:34.610 INFO:teuthology.orchestra.run.smithi098.stdout:2468 write 0x1311684 thru 0x1316c94 (0x5611 bytes) 2024-09-16T13:17:34.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:34.621 INFO:teuthology.orchestra.run.smithi098.stdout:2470 punch from 0x209db54 to 0x20a256b, (0x4a17 bytes) 2024-09-16T13:17:34.621 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.621 INFO:teuthology.orchestra.run.smithi098.stdout:2472 write 0x73d3e74 thru 0x73daefd (0x708a bytes) 2024-09-16T13:17:34.626 INFO:teuthology.orchestra.run.smithi098.stdout:2473 writesame 0xb4cd84 thru 0xb5c679 data_size 0xb(0xf8f6 bytes) 2024-09-16T13:17:34.637 INFO:teuthology.orchestra.run.smithi098.stdout:2474 writesame 0x3cae7d6 thru 0x3cbdc00 data_size 0x1(0xf42b bytes) 2024-09-16T13:17:34.689 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:34.679+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:34.696 INFO:teuthology.orchestra.run.smithi098.stdout:2477 trunc from 0xac5a293 to 0xe4fb502 2024-09-16T13:17:34.708 INFO:teuthology.orchestra.run.smithi098.stdout:2478 read 0x28f620d thru 0x2905178 (0xef6c bytes) 2024-09-16T13:17:34.718 INFO:teuthology.orchestra.run.smithi098.stdout:2479 trunc from 0xe4fb502 to 0xa3b446a 2024-09-16T13:17:34.724 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:34.724 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:34.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:34.746 INFO:teuthology.orchestra.run.smithi098.stdout:2480 read 0x940b47d thru 0x941a716 (0xf29a bytes) 2024-09-16T13:17:34.766 INFO:teuthology.orchestra.run.smithi098.stdout:2481 write 0xda06f4d thru 0xda16d7f (0xfe33 bytes) 2024-09-16T13:17:34.774 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.774 INFO:teuthology.orchestra.run.smithi098.stdout:2484 punch from 0x780c07a to 0x7812868, (0x67ee bytes) 2024-09-16T13:17:34.774 INFO:teuthology.orchestra.run.smithi098.stdout:2486 punch from 0x48c18c8 to 0x48c8963, (0x709b bytes) 2024-09-16T13:17:34.774 INFO:teuthology.orchestra.run.smithi098.stdout:2487 read 0xb0c32ae thru 0xb0cf5a7 (0xc2fa bytes) 2024-09-16T13:17:34.774 INFO:teuthology.orchestra.run.smithi098.stdout:2488 punch from 0x511779c to 0x5121393, (0x9bf7 bytes) 2024-09-16T13:17:34.774 INFO:teuthology.orchestra.run.smithi098.stdout:2490 read 0x9969347 thru 0x9969ee9 (0xba3 bytes) 2024-09-16T13:17:34.774 INFO:teuthology.orchestra.run.smithi098.stdout:2492 punch from 0x587ad4c to 0x587f2d8, (0x458c bytes) 2024-09-16T13:17:34.774 INFO:teuthology.orchestra.run.smithi098.stdout:2493 write 0xb9ea2fc thru 0xb9f9879 (0xf57e bytes) 2024-09-16T13:17:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:2494 read 0x2d47f86 thru 0x2d54265 (0xc2e0 bytes) 2024-09-16T13:17:34.780 INFO:teuthology.orchestra.run.smithi098.stdout:2495 write 0x8a3e641 thru 0x8a488da (0xa29a bytes) 2024-09-16T13:17:34.785 INFO:teuthology.orchestra.run.smithi098.stdout:2496 write 0x44e30f9 thru 0x44ebc0a (0x8b12 bytes) 2024-09-16T13:17:34.789 INFO:teuthology.orchestra.run.smithi098.stdout:2498 read 0xa79571f thru 0xa79950e (0x3df0 bytes) 2024-09-16T13:17:34.789 INFO:teuthology.orchestra.run.smithi098.stdout:2499 read 0xa4a440c thru 0xa4adc8a (0x987f bytes) 2024-09-16T13:17:34.789 INFO:teuthology.orchestra.run.smithi098.stdout:2500 punch from 0x8c58eaa to 0x8c6490f, (0xba65 bytes) 2024-09-16T13:17:34.789 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:34.779+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:34.796 INFO:teuthology.orchestra.run.smithi098.stdout:2502 trunc from 0xda16d80 to 0xa1b0e04 2024-09-16T13:17:34.823 INFO:teuthology.orchestra.run.smithi098.stdout:2503 trunc from 0xa1b0e04 to 0x8564706 2024-09-16T13:17:34.837 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:34.837 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:34.846 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.846 INFO:teuthology.orchestra.run.smithi098.stdout:2505 read 0x3f6362a thru 0x3f72093 (0xea6a bytes) 2024-09-16T13:17:34.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:34.855 INFO:teuthology.orchestra.run.smithi098.stdout:2506 trunc from 0x8564706 to 0x513c63d 2024-09-16T13:17:34.879 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.879 INFO:teuthology.orchestra.run.smithi098.stdout:2508 punch from 0x148a97f to 0x14993ff, (0xea80 bytes) 2024-09-16T13:17:34.879 INFO:teuthology.orchestra.run.smithi098.stdout:2509 read 0x496c403 thru 0x4977a4e (0xb64c bytes) 2024-09-16T13:17:34.879 INFO:teuthology.orchestra.run.smithi098.stdout:2513 write 0x23f12fa thru 0x2400c58 (0xf95f bytes) 2024-09-16T13:17:34.893 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:34.887+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:34.897 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.897 INFO:teuthology.orchestra.run.smithi098.stdout:2516 punch from 0x2c2536d to 0x2c2f57a, (0xa20d bytes) 2024-09-16T13:17:34.926 INFO:teuthology.orchestra.run.smithi098.stdout:2518 write 0xc479a8d thru 0xc487761 (0xdcd5 bytes) 2024-09-16T13:17:34.935 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:34.951 INFO:teuthology.orchestra.run.smithi098.stdout:2520 trunc from 0xc487762 to 0x3e3ab37 2024-09-16T13:17:34.976 INFO:teuthology.orchestra.run.smithi098.stdout:2521 punch from 0x2dbb209 to 0x2dc3fe9, (0x8de0 bytes) 2024-09-16T13:17:34.976 INFO:teuthology.orchestra.run.smithi098.stdout:2522 read 0x2b1a911 thru 0x2b2338b (0x8a7b bytes) 2024-09-16T13:17:34.977 INFO:teuthology.orchestra.run.smithi098.stdout:2523 punch from 0x2d01d36 to 0x2d119db, (0xfca5 bytes) 2024-09-16T13:17:34.977 INFO:teuthology.orchestra.run.smithi098.stdout:2524 write 0x2c153ce thru 0x2c1af23 (0x5b56 bytes) 2024-09-16T13:17:34.982 INFO:teuthology.orchestra.run.smithi098.stdout:2525 read 0x340e16d thru 0x341acd4 (0xcb68 bytes) 2024-09-16T13:17:34.992 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:34.988+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:34.994 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:34.994 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:35.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:35.006 INFO:teuthology.orchestra.run.smithi098.stdout:2526 write 0x807e636 thru 0x8081271 (0x2c3c bytes) 2024-09-16T13:17:35.039 INFO:teuthology.orchestra.run.smithi098.stdout:2527 writesame 0xd805633 thru 0xd805d08 data_size 0x19(0x6d6 bytes) 2024-09-16T13:17:35.039 INFO:teuthology.orchestra.run.smithi098.stdout:2528 writesame 0x90ed3a thru 0x919c13 data_size 0x1(0xaeda bytes) 2024-09-16T13:17:35.082 INFO:teuthology.orchestra.run.smithi098.stdout:2529 punch from 0x4a992f9 to 0x4aa5abf, (0xc7c6 bytes) 2024-09-16T13:17:35.082 INFO:teuthology.orchestra.run.smithi098.stdout:2531 write 0x1ddbd41 thru 0x1de96ce (0xd98e bytes) 2024-09-16T13:17:35.093 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:35.088+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:35.095 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:35.095 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:35.100 INFO:teuthology.orchestra.run.smithi098.stdout:2532 trunc from 0xd805d09 to 0xa027501 2024-09-16T13:17:35.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:35.115 INFO:teuthology.orchestra.run.smithi098.stdout:2533 writesame 0x7ded8e6 thru 0x7df660f data_size 0x26(0x8d2a bytes) 2024-09-16T13:17:35.123 INFO:teuthology.orchestra.run.smithi098.stdout:2534 read 0x832a08 thru 0x8402e8 (0xd8e1 bytes) 2024-09-16T13:17:35.125 INFO:teuthology.orchestra.run.smithi098.stdout:2535 read 0x69ba115 thru 0x69bbbfd (0x1ae9 bytes) 2024-09-16T13:17:35.125 INFO:teuthology.orchestra.run.smithi098.stdout:2536 punch from 0x189c1f9 to 0x18a0284, (0x408b bytes) 2024-09-16T13:17:35.125 INFO:teuthology.orchestra.run.smithi098.stdout:2537 writesame 0xd21d87 thru 0xd30de8 data_size 0x1d(0xf062 bytes) 2024-09-16T13:17:35.132 INFO:teuthology.orchestra.run.smithi098.stdout:2538 write 0x558720e thru 0x558e513 (0x7306 bytes) 2024-09-16T13:17:35.138 INFO:teuthology.orchestra.run.smithi098.stdout:2539 read 0x36c0e1 thru 0x36d319 (0x1239 bytes) 2024-09-16T13:17:35.139 INFO:teuthology.orchestra.run.smithi098.stdout:2540 read 0x9b7507c thru 0x9b7e1bf (0x9144 bytes) 2024-09-16T13:17:35.139 INFO:teuthology.orchestra.run.smithi098.stdout:2541 punch from 0x40da5ac to 0x40e27d5, (0x8229 bytes) 2024-09-16T13:17:35.139 INFO:teuthology.orchestra.run.smithi098.stdout:2542 write 0x71b2288 thru 0x71bd6ea (0xb463 bytes) 2024-09-16T13:17:35.145 INFO:teuthology.orchestra.run.smithi098.stdout:2543 write 0x6cecf11 thru 0x6cfc642 (0xf732 bytes) 2024-09-16T13:17:35.152 INFO:teuthology.orchestra.run.smithi098.stdout:2544 write 0x7977cec thru 0x797d658 (0x596d bytes) 2024-09-16T13:17:35.159 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:35.159 INFO:teuthology.orchestra.run.smithi098.stdout:2549 read 0x6558f6a thru 0x655b20f (0x22a6 bytes) 2024-09-16T13:17:35.159 INFO:teuthology.orchestra.run.smithi098.stdout:2550 read 0x9561ed8 thru 0x956cd82 (0xaeab bytes) 2024-09-16T13:17:35.159 INFO:teuthology.orchestra.run.smithi098.stdout:2551 punch from 0x55566f8 to 0x5564a94, (0xe39c bytes) 2024-09-16T13:17:35.160 INFO:teuthology.orchestra.run.smithi098.stdout:2552 write 0x49e44ae thru 0x49ea423 (0x5f76 bytes) 2024-09-16T13:17:35.166 INFO:teuthology.orchestra.run.smithi098.stdout:2553 read 0x27d1cf5 thru 0x27d7853 (0x5b5f bytes) 2024-09-16T13:17:35.167 INFO:teuthology.orchestra.run.smithi098.stdout:2554 write 0x8bdabe6 thru 0x8bdfebc (0x52d7 bytes) 2024-09-16T13:17:35.173 INFO:teuthology.orchestra.run.smithi098.stdout:2555 read 0x1d38c89 thru 0x1d45c47 (0xcfbf bytes) 2024-09-16T13:17:35.174 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:35.174 INFO:teuthology.orchestra.run.smithi098.stdout:2557 punch from 0x431deca to 0x4322a4f, (0x4b85 bytes) 2024-09-16T13:17:35.174 INFO:teuthology.orchestra.run.smithi098.stdout:2558 punch from 0x8abe0de to 0x8ac449f, (0x63c1 bytes) 2024-09-16T13:17:35.188 INFO:teuthology.orchestra.run.smithi098.stdout:2560 trunc from 0xa027501 to 0x4474d2e 2024-09-16T13:17:35.194 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:35.188+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:35.205 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:35.205 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:35.210 INFO:teuthology.orchestra.run.smithi098.stdout:2561 punch from 0x1f0d0fc to 0x1f12fe8, (0x5eec bytes) 2024-09-16T13:17:35.210 INFO:teuthology.orchestra.run.smithi098.stdout:2562 read 0x2d2a736 thru 0x2d31977 (0x7242 bytes) 2024-09-16T13:17:35.210 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:35.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:35.239 INFO:teuthology.orchestra.run.smithi098.stdout:2565 write 0xd6be9a6 thru 0xd6c22b8 (0x3913 bytes) 2024-09-16T13:17:35.245 INFO:teuthology.orchestra.run.smithi098.stdout:2567 read 0x167f9b2 thru 0x1687cda (0x8329 bytes) 2024-09-16T13:17:35.245 INFO:teuthology.orchestra.run.smithi098.stdout:2568 write 0x9f1707a thru 0x9f1d41c (0x63a3 bytes) 2024-09-16T13:17:35.262 INFO:teuthology.orchestra.run.smithi098.stdout:2569 trunc from 0xd6c22b9 to 0x74bd770 2024-09-16T13:17:35.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:17:35.282 INFO:teuthology.orchestra.run.smithi098.stdout:2570 writesame 0x6863a0a thru 0x686cd40 data_size 0xd(0x9337 bytes) 2024-09-16T13:17:35.291 INFO:teuthology.orchestra.run.smithi098.stdout:2571 read 0x101139a thru 0x10125d7 (0x123e bytes) 2024-09-16T13:17:35.292 INFO:teuthology.orchestra.run.smithi098.stdout:2572 punch from 0x62c04 to 0x669c7, (0x3dc3 bytes) 2024-09-16T13:17:35.294 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:35.288+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:35.309 INFO:teuthology.orchestra.run.smithi098.stdout:2575 write 0x9497c70 thru 0x94a0035 (0x83c6 bytes) 2024-09-16T13:17:35.313 INFO:teuthology.orchestra.run.smithi098.stdout:2576 writesame 0x4407254 thru 0x440bc1a data_size 0x11(0x49c7 bytes) 2024-09-16T13:17:35.321 INFO:teuthology.orchestra.run.smithi098.stdout:2579 read 0x881f815 thru 0x882afa7 (0xb793 bytes) 2024-09-16T13:17:35.322 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:35.322 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:35.328 INFO:teuthology.orchestra.run.smithi098.stdout:2580 trunc from 0x94a0036 to 0x66edab7 2024-09-16T13:17:35.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:35.353 INFO:teuthology.orchestra.run.smithi098.stdout:2581 trunc from 0x66edab7 to 0x9d5524a 2024-09-16T13:17:35.363 INFO:teuthology.orchestra.run.smithi098.stdout:2582 flatten 2024-09-16T13:17:35.395 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:35.392+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:35.397 INFO:teuthology.orchestra.run.smithi098.stdout:2584 read 0x20fd6e4 thru 0x21061fc (0x8b19 bytes) 2024-09-16T13:17:35.398 INFO:teuthology.orchestra.run.smithi098.stdout:2585 write 0x2de790d thru 0x2df02cb (0x89bf bytes) 2024-09-16T13:17:35.403 INFO:teuthology.orchestra.run.smithi098.stdout:2586 punch from 0x7df4d6f to 0x7e012eb, (0xc57c bytes) 2024-09-16T13:17:35.403 INFO:teuthology.orchestra.run.smithi098.stdout:2587 read 0x597c43e thru 0x5984ee8 (0x8aab bytes) 2024-09-16T13:17:35.413 INFO:teuthology.orchestra.run.smithi098.stdout:2588 trunc from 0x9d5524a to 0x494e519 2024-09-16T13:17:35.436 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:35.506 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:35.508 INFO:teuthology.orchestra.run.smithi098.stdout:2589 write 0x6882838 thru 0x6888f70 (0x6739 bytes) 2024-09-16T13:17:35.508 INFO:teuthology.orchestra.run.smithi098.stdout:2590 trunc from 0x6888f71 to 0xc3c4ccf 2024-09-16T13:17:35.508 INFO:teuthology.orchestra.run.smithi098.stdout:2592 trunc from 0xc3c4ccf to 0x50470f0 2024-09-16T13:17:35.508 INFO:teuthology.orchestra.run.smithi098.stdout:2593 punch from 0x1b29697 to 0x1b34934, (0xb29d bytes) 2024-09-16T13:17:35.508 INFO:teuthology.orchestra.run.smithi098.stdout:2594 read 0x3c5ea1e thru 0x3c636a8 (0x4c8b bytes) 2024-09-16T13:17:35.508 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:35.504+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:35.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:35.526 INFO:teuthology.orchestra.run.smithi098.stdout:2595 write 0xbf7784f thru 0xbf83fb8 (0xc76a bytes) 2024-09-16T13:17:35.534 INFO:teuthology.orchestra.run.smithi098.stdout:2597 read 0x1542912 thru 0x154975c (0x6e4b bytes) 2024-09-16T13:17:35.534 INFO:teuthology.orchestra.run.smithi098.stdout:2598 punch from 0x437308c to 0x43756f4, (0x2668 bytes) 2024-09-16T13:17:35.534 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:35.534 INFO:teuthology.orchestra.run.smithi098.stdout:2600 read 0x6f404f6 thru 0x6f4e5ae (0xe0b9 bytes) 2024-09-16T13:17:35.534 INFO:teuthology.orchestra.run.smithi098.stdout:2601 writesame 0x899adce thru 0x899f366 data_size 0x1(0x4599 bytes) 2024-09-16T13:17:35.558 INFO:teuthology.orchestra.run.smithi098.stdout:2602 write 0x6cbedaf thru 0x6cbfbfb (0xe4d bytes) 2024-09-16T13:17:35.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:35.564 INFO:teuthology.orchestra.run.smithi098.stdout:2603 write 0x940f141 thru 0x9414e9e (0x5d5e bytes) 2024-09-16T13:17:35.571 INFO:teuthology.orchestra.run.smithi098.stdout:2604 write 0x5699c24 thru 0x569a8b0 (0xc8d bytes) 2024-09-16T13:17:35.588 INFO:teuthology.orchestra.run.smithi098.stdout:2605 write 0xc9cddf8 thru 0xc9d4040 (0x6249 bytes) 2024-09-16T13:17:35.593 INFO:teuthology.orchestra.run.smithi098.stdout:2607 write 0xac07fd4 thru 0xac14442 (0xc46f bytes) 2024-09-16T13:17:35.599 INFO:teuthology.orchestra.run.smithi098.stdout:2608 write 0x12993ed thru 0x129e726 (0x533a bytes) 2024-09-16T13:17:35.602 INFO:teuthology.orchestra.run.smithi098.stdout:2609 read 0xb96a058 thru 0xb96b89d (0x1846 bytes) 2024-09-16T13:17:35.602 INFO:teuthology.orchestra.run.smithi098.stdout:2610 writesame 0x6b16733 thru 0x6b17230 data_size 0x7(0xafe bytes) 2024-09-16T13:17:35.607 INFO:teuthology.orchestra.run.smithi098.stdout:2611 read 0x2aec640 thru 0x2af389d (0x725e bytes) 2024-09-16T13:17:35.608 INFO:teuthology.orchestra.run.smithi098.stdout:2613 read 0x9de1f28 thru 0x9de3db1 (0x1e8a bytes) 2024-09-16T13:17:35.608 INFO:teuthology.orchestra.run.smithi098.stdout:2614 write 0x911b888 thru 0x9127f25 (0xc69e bytes) 2024-09-16T13:17:35.608 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:35.604+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:35.624 INFO:teuthology.orchestra.run.smithi098.stdout:2615 write 0xdcff9b6 thru 0xdd08510 (0x8b5b bytes) 2024-09-16T13:17:35.629 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:35.629 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:35.631 INFO:teuthology.orchestra.run.smithi098.stdout:2616 write 0xa4f2010 thru 0xa4ffe40 (0xde31 bytes) 2024-09-16T13:17:35.636 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:35.636 INFO:teuthology.orchestra.run.smithi098.stdout:2618 write 0xc95f74e thru 0xc96c5b0 (0xce63 bytes) 2024-09-16T13:17:35.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:35.642 INFO:teuthology.orchestra.run.smithi098.stdout:2620 write 0x919a9b1 thru 0x91a71cc (0xc81c bytes) 2024-09-16T13:17:35.648 INFO:teuthology.orchestra.run.smithi098.stdout:2621 write 0x3b401b4 thru 0x3b4df62 (0xddaf bytes) 2024-09-16T13:17:35.654 INFO:teuthology.orchestra.run.smithi098.stdout:2624 read 0x3f05431 thru 0x3f141b2 (0xed82 bytes) 2024-09-16T13:17:35.677 INFO:teuthology.orchestra.run.smithi098.stdout:2625 trunc from 0xdd08511 to 0x1a804c2 2024-09-16T13:17:35.702 INFO:teuthology.orchestra.run.smithi098.stdout:2626 read 0x169b735 thru 0x169fa27 (0x42f3 bytes) 2024-09-16T13:17:35.703 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:35.708 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:35.704+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:35.733 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:35.733 INFO:teuthology.orchestra.run.smithi098.stdout:2627 write 0xe40471b thru 0xe40b617 (0x6efd bytes) 2024-09-16T13:17:35.735 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:35.735 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:35.738 INFO:teuthology.orchestra.run.smithi098.stdout:2628 write 0x2a8ee70 thru 0x2a96908 (0x7a99 bytes) 2024-09-16T13:17:35.744 INFO:teuthology.orchestra.run.smithi098.stdout:2629 punch from 0xac06173 to 0xac07857, (0x16e4 bytes) 2024-09-16T13:17:35.744 INFO:teuthology.orchestra.run.smithi098.stdout:2631 writesame 0x771e48b thru 0x771e8fe data_size 0x14(0x474 bytes) 2024-09-16T13:17:35.748 INFO:teuthology.orchestra.run.smithi098.stdout:2632 write 0x4d437d3 thru 0x4d45899 (0x20c7 bytes) 2024-09-16T13:17:35.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:35.752 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:35.753 INFO:teuthology.orchestra.run.smithi098.stdout:2633 read 0x4adb177 thru 0x4ae9446 (0xe2d0 bytes) 2024-09-16T13:17:35.753 INFO:teuthology.orchestra.run.smithi098.stdout:2634 punch from 0x4d2647c to 0x4d29698, (0x321c bytes) 2024-09-16T13:17:35.753 INFO:teuthology.orchestra.run.smithi098.stdout:2635 write 0x1821cc4 thru 0x183161d (0xf95a bytes) 2024-09-16T13:17:35.756 INFO:teuthology.orchestra.run.smithi098.stdout:2636 read 0x1269292 thru 0x126cd99 (0x3b08 bytes) 2024-09-16T13:17:35.758 INFO:teuthology.orchestra.run.smithi098.stdout:2639 trunc from 0xe40b618 to 0xee02a5d 2024-09-16T13:17:35.767 INFO:teuthology.orchestra.run.smithi098.stdout:2640 writesame 0xe641eec thru 0xe649d88 data_size 0x1(0x7e9d bytes) 2024-09-16T13:17:35.777 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:2641 punch from 0x4c5ec71 to 0x4c62a07, (0x3d96 bytes) 2024-09-16T13:17:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:2642 write 0xe49f469 thru 0xe4a1571 (0x2109 bytes) 2024-09-16T13:17:35.801 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:35.802 INFO:teuthology.orchestra.run.smithi098.stdout:2643 write 0x8842ed1 thru 0x8847255 (0x4385 bytes) 2024-09-16T13:17:35.809 INFO:teuthology.orchestra.run.smithi098.stdout:2644 read 0x15623c thru 0x15ca5f (0x6824 bytes) 2024-09-16T13:17:35.809 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:35.804+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:35.810 INFO:teuthology.orchestra.run.smithi098.stdout:2645 punch from 0x35715e4 to 0x35737d4, (0x21f0 bytes) 2024-09-16T13:17:35.810 INFO:teuthology.orchestra.run.smithi098.stdout:2646 read 0xadab878 thru 0xadba093 (0xe81c bytes) 2024-09-16T13:17:35.822 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:35.830 INFO:teuthology.orchestra.run.smithi098.stdout:2647 trunc from 0xee02a5d to 0x4a3702d 2024-09-16T13:17:35.844 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:35.853 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:35.853 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:35.853 INFO:teuthology.orchestra.run.smithi098.stdout:2650 read 0x9a0234 thru 0x9a4da5 (0x4b72 bytes) 2024-09-16T13:17:35.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:35.865 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:35.867 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:35.867 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:35.867 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:35.867 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:35.867 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:35.867 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:35.868 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:35.868 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:35.868 INFO:teuthology.orchestra.run.smithi098.stdout:2652 trunc from 0x4a3702d to 0xbe2edab 2024-09-16T13:17:35.877 INFO:teuthology.orchestra.run.smithi098.stdout:2653 read 0x9326e5b thru 0x932988c (0x2a32 bytes) 2024-09-16T13:17:35.892 INFO:teuthology.orchestra.run.smithi098.stdout:2654 trunc from 0xbe2edab to 0x33279c3 2024-09-16T13:17:35.904 INFO:teuthology.orchestra.run.smithi098.stdout:2658 read 0x261f4b2 thru 0x26202ae (0xdfd bytes) 2024-09-16T13:17:35.910 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:35.904+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:35.920 INFO:teuthology.orchestra.run.smithi098.stdout:2659 writesame 0x45efb78 thru 0x45f1406 data_size 0x1(0x188f bytes) 2024-09-16T13:17:35.939 INFO:teuthology.orchestra.run.smithi098.stdout:2660 trunc from 0x45f1407 to 0xd9af967 2024-09-16T13:17:35.954 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:35.954 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:35.954 INFO:teuthology.orchestra.run.smithi098.stdout:2667 punch from 0x379f2c1 to 0x37aa790, (0xb4cf bytes) 2024-09-16T13:17:35.954 INFO:teuthology.orchestra.run.smithi098.stdout:2670 read 0x3623e76 thru 0x362979f (0x592a bytes) 2024-09-16T13:17:35.954 INFO:teuthology.orchestra.run.smithi098.stdout:2672 read 0x20c5ad thru 0x2129e0 (0x6434 bytes) 2024-09-16T13:17:35.955 INFO:teuthology.orchestra.run.smithi098.stdout:2673 write 0x9ca2ba6 thru 0x9ca7c2f (0x508a bytes) 2024-09-16T13:17:35.960 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:35.960 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:35.961 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:35.961 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:35.961 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:35.961 INFO:teuthology.orchestra.run.smithi098.stdout:2674 write 0xab26c0c thru 0xab3198f (0xad84 bytes) 2024-09-16T13:17:35.969 INFO:teuthology.orchestra.run.smithi098.stdout:2677 write 0x4406476 thru 0x440d1dc (0x6d67 bytes) 2024-09-16T13:17:35.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:35.975 INFO:teuthology.orchestra.run.smithi098.stdout:2678 read 0x26b9c45 thru 0x26c6301 (0xc6bd bytes) 2024-09-16T13:17:35.975 INFO:teuthology.orchestra.run.smithi098.stdout:2680 write 0xafbe298 thru 0xafc90cf (0xae38 bytes) 2024-09-16T13:17:35.980 INFO:teuthology.orchestra.run.smithi098.stdout:2681 write 0x8f43135 thru 0x8f4e3bd (0xb289 bytes) 2024-09-16T13:17:35.986 INFO:teuthology.orchestra.run.smithi098.stdout:2682 write 0x550ffca thru 0x55194a3 (0x94da bytes) 2024-09-16T13:17:35.992 INFO:teuthology.orchestra.run.smithi098.stdout:2685 write 0x12c8f6e thru 0x12d33f7 (0xa48a bytes) 2024-09-16T13:17:35.995 INFO:teuthology.orchestra.run.smithi098.stdout:2686 write 0x7657629 thru 0x7664f69 (0xd941 bytes) 2024-09-16T13:17:36.011 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:36.004+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:36.013 INFO:teuthology.orchestra.run.smithi098.stdout:2687 trunc from 0xd9af967 to 0x8cf6e9e 2024-09-16T13:17:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:2689 read 0x2f413c2 thru 0x2f46845 (0x5484 bytes) 2024-09-16T13:17:36.053 INFO:teuthology.orchestra.run.smithi098.stdout:2690 writesame 0xadb6377 thru 0xadc1139 data_size 0x1(0xadc3 bytes) 2024-09-16T13:17:36.053 INFO:teuthology.orchestra.run.smithi098.stdout:2691 punch from 0x58ce87f to 0x58de6d1, (0xfe52 bytes) 2024-09-16T13:17:36.053 INFO:teuthology.orchestra.run.smithi098.stdout:2692 read 0x24f813a thru 0x24fec65 (0x6b2c bytes) 2024-09-16T13:17:36.066 INFO:teuthology.orchestra.run.smithi098.stdout:2693 writesame 0xb4c200b thru 0xb4ca40f data_size 0x1(0x8405 bytes) 2024-09-16T13:17:36.071 INFO:teuthology.orchestra.run.smithi098.stdout:2694 trunc from 0xb4ca410 to 0xd48b470 2024-09-16T13:17:36.078 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:36.078 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:36.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:36.078 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:36.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.078 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:36.078 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:36.078 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.047194335, 2024-09-16T13:17:36.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:17:36.078 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:17:36.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:36.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.047046928999997, 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.084 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:17:36.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:17:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.047019585999998, 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:17:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.488344324000003, 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:17:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.488256732000004, 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:17:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:17:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:17:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:17:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:17:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:17:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.488086809999999, 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:17:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:17:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2259 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3014656~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336758234000001, 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016650893999999999, 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2259 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735831+0000", 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9530000000000005e-06 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735836+0000", 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7450000000000003e-06 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735839+0000", 2024-09-16T13:17:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.915e-06 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752296+0000", 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016457241000000001 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752360+0000", 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4108000000000004e-05 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752473+0000", 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112305 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2260 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336738312999998, 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016799073000000001, 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2260 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735843+0000", 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3480000000000001e-06 2024-09-16T13:17:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735847+0000", 2024-09-16T13:17:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5990000000000002e-06 2024-09-16T13:17:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735849+0000", 2024-09-16T13:17:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5859999999999999e-06 2024-09-16T13:17:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752509+0000", 2024-09-16T13:17:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016659377 2024-09-16T13:17:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752559+0000", 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9774000000000002e-05 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752641+0000", 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2057000000000004e-05 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2261 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336726124000002, 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016960698, 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2261 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.03e-07 2024-09-16T13:17:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735855+0000", 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735860+0000", 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1110000000000001e-06 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735862+0000", 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752695+0000", 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01683343 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752744+0000", 2024-09-16T13:17:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8414000000000001e-05 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752814+0000", 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0623000000000005e-05 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2262 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1048576~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336715509999998, 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017224107999999998, 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2262 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735865+0000", 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.91e-07 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735866+0000", 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1480000000000001e-06 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735870+0000", 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735877+0000", 2024-09-16T13:17:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4370000000000001e-06 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752871+0000", 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016994061000000001 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752962+0000", 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0470000000000006e-05 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753089+0000", 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126642 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2263 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336700051000001, 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017392088999999999, 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2263 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-07 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735881+0000", 2024-09-16T13:17:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-09-16T13:17:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735885+0000", 2024-09-16T13:17:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4180000000000001e-06 2024-09-16T13:17:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735887+0000", 2024-09-16T13:17:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-06 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753116+0000", 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017228963 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753184+0000", 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7374999999999998e-05 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753272+0000", 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8323000000000001e-05 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2264 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2064384~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.3366902, 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017630541999999999, 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2264 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735891+0000", 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735895+0000", 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735897+0000", 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4669999999999998e-06 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753316+0000", 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017418685 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753403+0000", 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7590000000000007e-05 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753520+0000", 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116863 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2265 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1081344~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336680317999999, 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017739263000000002, 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2265 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735901+0000", 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735904+0000", 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.304e-06 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735907+0000", 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3599999999999999e-06 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753534+0000", 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017627446000000001 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753576+0000", 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1474999999999998e-05 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753639+0000", 2024-09-16T13:17:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3188999999999994e-05 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2266 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2097152~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336671084999999, 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017970893000000002, 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2266 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735914+0000", 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1699999999999996e-06 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735918+0000", 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6859999999999999e-06 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735920+0000", 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4870000000000001e-06 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753685+0000", 2024-09-16T13:17:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017764582000000001 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753771+0000", 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6259000000000002e-05 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753880+0000", 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010844300000000001 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2267 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336657184000003, 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018110804000000001, 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2267 2024-09-16T13:17:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735924+0000", 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735928+0000", 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.416e-06 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735930+0000", 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753898+0000", 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017967852999999999 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753955+0000", 2024-09-16T13:17:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6662000000000001e-05 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754034+0000", 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8904000000000002e-05 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2268 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3080192~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336647698999997, 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018252866, 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2268 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735934+0000", 2024-09-16T13:17:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0950000000000001e-06 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735938+0000", 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735941+0000", 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8750000000000002e-06 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754071+0000", 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018129941 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754120+0000", 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8359000000000002e-05 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754185+0000", 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5358999999999995e-05 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2269 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336636319999997, 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018533516, 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2269 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735945+0000", 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1149999999999999e-06 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735948+0000", 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5250000000000001e-06 2024-09-16T13:17:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735951+0000", 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9909999999999998e-06 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754270+0000", 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018318081999999999 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754325+0000", 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5648e-05 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754477+0000", 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015188199999999999 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2270 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1114112~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336625851000001, 2024-09-16T13:17:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019103282999999999, 2024-09-16T13:17:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2270 2024-09-16T13:17:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735955+0000", 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735956+0000", 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.096e-06 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735959+0000", 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5140000000000002e-06 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735963+0000", 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.053e-06 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754822+0000", 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018858883 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754941+0000", 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118789 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755057+0000", 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116272 2024-09-16T13:17:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2271 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1146880~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336604039000001, 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019332309999999998, 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2271 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3500000000000002e-07 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735982+0000", 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735987+0000", 2024-09-16T13:17:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1930000000000003e-06 2024-09-16T13:17:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735993+0000", 2024-09-16T13:17:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4659999999999996e-06 2024-09-16T13:17:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755081+0000", 2024-09-16T13:17:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019088187 2024-09-16T13:17:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755194+0000", 2024-09-16T13:17:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011223299999999999 2024-09-16T13:17:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755308+0000", 2024-09-16T13:17:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114663 2024-09-16T13:17:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2272 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 163840~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.336582208999999, 2024-09-16T13:17:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019476291, 2024-09-16T13:17:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:36.183 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2272 2024-09-16T13:17:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:36.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:36.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:36.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:36.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:36.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:36.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:36.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:36.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:36.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736003+0000", 2024-09-16T13:17:36.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2279999999999998e-06 2024-09-16T13:17:36.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:36.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736007+0000", 2024-09-16T13:17:36.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.506e-06 2024-09-16T13:17:36.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736011+0000", 2024-09-16T13:17:36.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9280000000000003e-06 2024-09-16T13:17:36.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:36.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755336+0000", 2024-09-16T13:17:36.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019325189999999999 2024-09-16T13:17:36.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:36.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755393+0000", 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7293999999999997e-05 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755474+0000", 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0635000000000003e-05 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:36.190 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:36.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:2698 write 0xe02a70d thru 0xe02b84b (0x113f bytes) 2024-09-16T13:17:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:2699 writesame 0xde9153d thru 0xde92419 data_size 0x1(0xedd bytes) 2024-09-16T13:17:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:2700 write 0x231fc32 thru 0x2323fe3 (0x43b2 bytes) 2024-09-16T13:17:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:2702 read 0x35f4bde thru 0x35fb9b9 (0x6ddc bytes) 2024-09-16T13:17:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:2706 write 0x7ffcfe6 thru 0x7ffe899 (0x18b4 bytes) 2024-09-16T13:17:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:2707 read 0x8f63cc1 thru 0x8f6df2f (0xa26f bytes) 2024-09-16T13:17:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:2710 trunc from 0xe02b84c to 0xaf9b104 2024-09-16T13:17:36.194 INFO:teuthology.orchestra.run.smithi098.stdout:2711 write 0x8791167 thru 0x879b07c (0x9f16 bytes) 2024-09-16T13:17:36.194 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:36.188+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:36.195 INFO:teuthology.orchestra.run.smithi098.stdout:2712 trunc from 0xaf9b104 to 0x52c56 2024-09-16T13:17:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:2714 read 0xb54c thru 0x16f55 (0xba0a bytes) 2024-09-16T13:17:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:2715 punch from 0x221d2 to 0x2278d, (0x5bb bytes) 2024-09-16T13:17:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:2718 read 0x37291 thru 0x3959c (0x230c bytes) 2024-09-16T13:17:36.226 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.227 INFO:teuthology.orchestra.run.smithi098.stdout:2721 read 0xff7d thru 0x177c6 (0x784a bytes) 2024-09-16T13:17:36.247 INFO:teuthology.orchestra.run.smithi098.stdout:2722 trunc from 0x52c56 to 0xb87da2c 2024-09-16T13:17:36.261 INFO:teuthology.orchestra.run.smithi098.stdout:2723 writesame 0x6fa8ddd thru 0x6fb7286 data_size 0x1(0xe4aa bytes) 2024-09-16T13:17:36.292 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:36.288+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:36.298 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:36.299 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:36.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:36.311 INFO:teuthology.orchestra.run.smithi098.stdout:2724 read 0x2fb8c6e thru 0x2fb9880 (0xc13 bytes) 2024-09-16T13:17:36.311 INFO:teuthology.orchestra.run.smithi098.stdout:2725 write 0x797e09c thru 0x797fa31 (0x1996 bytes) 2024-09-16T13:17:36.320 INFO:teuthology.orchestra.run.smithi098.stdout:2726 read 0xb3a8809 thru 0xb3ae44e (0x5c46 bytes) 2024-09-16T13:17:36.320 INFO:teuthology.orchestra.run.smithi098.stdout:2728 read 0x3520104 thru 0x3521548 (0x1445 bytes) 2024-09-16T13:17:36.337 INFO:teuthology.orchestra.run.smithi098.stdout:2729 write 0xc61105a thru 0xc618f91 (0x7f38 bytes) 2024-09-16T13:17:36.346 INFO:teuthology.orchestra.run.smithi098.stdout:2731 trunc from 0xc618f92 to 0xdc1cdeb 2024-09-16T13:17:36.359 INFO:teuthology.orchestra.run.smithi098.stdout:2732 write 0x235f00e thru 0x236c302 (0xd2f5 bytes) 2024-09-16T13:17:36.366 INFO:teuthology.orchestra.run.smithi098.stdout:2733 read 0x2f2ea9b thru 0x2f34ae9 (0x604f bytes) 2024-09-16T13:17:36.366 INFO:teuthology.orchestra.run.smithi098.stdout:2734 write 0x9f0757f thru 0x9f083de (0xe60 bytes) 2024-09-16T13:17:36.378 INFO:teuthology.orchestra.run.smithi098.stdout:2735 trunc from 0xdc1cdeb to 0xb817077 2024-09-16T13:17:36.393 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:36.388+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:36.403 INFO:teuthology.orchestra.run.smithi098.stdout:2736 write 0x8bab059 thru 0x8baec5f (0x3c07 bytes) 2024-09-16T13:17:36.410 INFO:teuthology.orchestra.run.smithi098.stdout:2737 punch from 0x3efdc74 to 0x3f0b146, (0xd4d2 bytes) 2024-09-16T13:17:36.410 INFO:teuthology.orchestra.run.smithi098.stdout:2738 read 0xbfade thru 0xc483d (0x4d60 bytes) 2024-09-16T13:17:36.410 INFO:teuthology.orchestra.run.smithi098.stdout:2739 write 0x4232b8f thru 0x4242356 (0xf7c8 bytes) 2024-09-16T13:17:36.412 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:36.412 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:36.413 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:36.417 INFO:teuthology.orchestra.run.smithi098.stdout:2740 read 0x92a73c1 thru 0x92af909 (0x8549 bytes) 2024-09-16T13:17:36.418 INFO:teuthology.orchestra.run.smithi098.stdout:2741 write 0x16f1025 thru 0x1700386 (0xf362 bytes) 2024-09-16T13:17:36.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:36.426 INFO:teuthology.orchestra.run.smithi098.stdout:2742 read 0x40af7fd thru 0x40badc5 (0xb5c9 bytes) 2024-09-16T13:17:36.427 INFO:teuthology.orchestra.run.smithi098.stdout:2743 write 0xa0bc5a7 thru 0xa0bed63 (0x27bd bytes) 2024-09-16T13:17:36.433 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.438 INFO:teuthology.orchestra.run.smithi098.stdout:2745 trunc from 0xb817077 to 0x9762478 2024-09-16T13:17:36.460 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.460 INFO:teuthology.orchestra.run.smithi098.stdout:2748 punch from 0x8255809 to 0x8258981, (0x3178 bytes) 2024-09-16T13:17:36.460 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.460 INFO:teuthology.orchestra.run.smithi098.stdout:2750 punch from 0x6119cd3 to 0x611bc4a, (0x1f77 bytes) 2024-09-16T13:17:36.461 INFO:teuthology.orchestra.run.smithi098.stdout:2751 read 0x26afe59 thru 0x26bf2dd (0xf485 bytes) 2024-09-16T13:17:36.461 INFO:teuthology.orchestra.run.smithi098.stdout:2752 punch from 0x79ce19d to 0x79d0a62, (0x28c5 bytes) 2024-09-16T13:17:36.461 INFO:teuthology.orchestra.run.smithi098.stdout:2753 punch from 0x8db13af to 0x8dbc605, (0xb256 bytes) 2024-09-16T13:17:36.461 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.461 INFO:teuthology.orchestra.run.smithi098.stdout:2755 read 0x357e39e thru 0x358b03f (0xcca2 bytes) 2024-09-16T13:17:36.461 INFO:teuthology.orchestra.run.smithi098.stdout:2757 writesame 0x7a8015a thru 0x7a8fa21 data_size 0x26(0xf8c8 bytes) 2024-09-16T13:17:36.477 INFO:teuthology.orchestra.run.smithi098.stdout:2759 trunc from 0x9762478 to 0x4abe59c 2024-09-16T13:17:36.494 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:36.487+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:36.502 INFO:teuthology.orchestra.run.smithi098.stdout:2760 write 0x1a93aa8 thru 0x1a9a351 (0x68aa bytes) 2024-09-16T13:17:36.509 INFO:teuthology.orchestra.run.smithi098.stdout:2761 read 0x1ad557a thru 0x1ae02f6 (0xad7d bytes) 2024-09-16T13:17:36.510 INFO:teuthology.orchestra.run.smithi098.stdout:2763 read 0x1dc63b0 thru 0x1dcd3b7 (0x7008 bytes) 2024-09-16T13:17:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:17:36.543 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11500 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 356352~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.041682432999998, 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014537983000000001, 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11500 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496474+0000", 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.801e-06 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496480+0000", 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1140000000000002e-06 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496484+0000", 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8770000000000003e-06 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507573+0000", 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011088473 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507624+0000", 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1591999999999997e-05 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511010+0000", 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033857380000000001 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11501 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 430080~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.041666066000001, 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014726846, 2024-09-16T13:17:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11501 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496489+0000", 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3700000000000001e-07 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496491+0000", 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9570000000000001e-06 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496497+0000", 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2550000000000003e-06 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496501+0000", 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8639999999999998e-06 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511060+0000", 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014559462 2024-09-16T13:17:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511125+0000", 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4399999999999993e-05 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511215+0000", 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0470999999999995e-05 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11530 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 339968~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.040731877999999, 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014141661, 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11530 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497423+0000", 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-07 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497434+0000", 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1195e-05 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497442+0000", 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.323e-06 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497446+0000", 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1049999999999997e-06 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511275+0000", 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013828872000000001 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511389+0000", 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114493 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511564+0000", 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017508700000000001 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11533 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 413696~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.040702794000001, 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014296237999999999, 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11533 2024-09-16T13:17:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.51e-07 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497458+0000", 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9139999999999996e-06 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497464+0000", 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0469999999999999e-06 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497467+0000", 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5030000000000002e-06 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511593+0000", 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014125692 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511669+0000", 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5950000000000003e-05 2024-09-16T13:17:36.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511748+0000", 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8681000000000001e-05 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11551 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 335872~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.040646766999998, 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014415314, 2024-09-16T13:17:36.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11551 2024-09-16T13:17:36.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497510+0000", 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.012e-06 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497516+0000", 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7880000000000003e-06 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497524+0000", 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9049999999999997e-06 2024-09-16T13:17:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511769+0000", 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245809 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511818+0000", 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8092e-05 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511923+0000", 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010532299999999999 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11553 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 413696~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.040623918000001, 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014561233999999999, 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.567 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11553 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497531+0000", 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1899999999999998e-07 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497533+0000", 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9149999999999999e-06 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497539+0000", 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.088e-06 2024-09-16T13:17:36.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497550+0000", 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0834e-05 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511962+0000", 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411947 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512018+0000", 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6755999999999997e-05 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512092+0000", 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3275000000000006e-05 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11576 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 335872~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499621+0000", 2024-09-16T13:17:36.571 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.038532942, 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013787382000000001, 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11576 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499621+0000", 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499621+0000", 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499622+0000", 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6400000000000003e-07 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499624+0000", 2024-09-16T13:17:36.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499630+0000", 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9429999999999997e-06 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499634+0000", 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513288+0000", 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013654463 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513342+0000", 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4357000000000002e-05 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513409+0000", 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6456000000000002e-05 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.575 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11577 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 409600~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.038516674, 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013920979999999999, 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.576 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11577 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499640+0000", 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1430000000000001e-06 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499646+0000", 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1029999999999998e-06 2024-09-16T13:17:36.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499650+0000", 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6370000000000001e-06 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513442+0000", 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013791805000000001 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513494+0000", 2024-09-16T13:17:36.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2074999999999999e-05 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513559+0000", 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4874000000000003e-05 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11628 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 471040~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.038159480000001, 2024-09-16T13:17:36.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013902296999999999, 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11628 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3599999999999999e-07 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499997+0000", 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.181e-06 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500004+0000", 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4609999999999996e-06 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500008+0000", 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0629999999999999e-06 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513576+0000", 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013567818000000001 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513626+0000", 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0013999999999999e-05 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513897+0000", 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027132399999999998 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11637 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.037624243, 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013965546000000001, 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11637 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500531+0000", 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2699999999999999e-07 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500534+0000", 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.055e-06 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500542+0000", 2024-09-16T13:17:36.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6389999999999998e-06 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500553+0000", 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0974e-05 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514249+0000", 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013696086 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514308+0000", 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8269000000000001e-05 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514496+0000", 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018799600000000001 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11638 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:36.589 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.037594761000001, 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014058684, 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11638 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:36.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500563+0000", 2024-09-16T13:17:36.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0790000000000001e-06 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500580+0000", 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6472000000000001e-05 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500586+0000", 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2199999999999997e-06 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514522+0000", 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013935774 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514564+0000", 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2549000000000002e-05 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514618+0000", 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3942e-05 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.593 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11642 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.037509927999999, 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014093902, 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11642 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.594 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500645+0000", 2024-09-16T13:17:36.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5200000000000002e-07 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500656+0000", 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1372999999999999e-05 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500664+0000", 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9019999999999999e-06 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500677+0000", 2024-09-16T13:17:36.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2954e-05 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514649+0000", 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013971417 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514692+0000", 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3154000000000001e-05 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514738+0000", 2024-09-16T13:17:36.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6650000000000002e-05 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11646 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.037471665999998, 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245793, 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11646 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500685+0000", 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2019999999999998e-06 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500698+0000", 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.279e-05 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500702+0000", 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7629999999999998e-06 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514772+0000", 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014070384999999999 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514865+0000", 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2980999999999996e-05 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514928+0000", 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3114000000000006e-05 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11648 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.037448535999999, 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411604999999999, 2024-09-16T13:17:36.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11648 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9900000000000001e-07 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500713+0000", 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.793e-06 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500719+0000", 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500722+0000", 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3809999999999999e-06 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514973+0000", 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014250924 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515031+0000", 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7864e-05 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515117+0000", 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6697e-05 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11654 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.037081701999998, 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014438998, 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11654 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8699999999999995e-07 2024-09-16T13:17:36.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501075+0000", 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501082+0000", 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.844e-06 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501086+0000", 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7749999999999999e-06 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515152+0000", 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014065891000000001 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515232+0000", 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0461000000000001e-05 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515512+0000", 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027934599999999998 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11675 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.036619108, 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014239586, 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.611 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11675 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:36.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501536+0000", 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501539+0000", 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3529999999999999e-06 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501546+0000", 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.109e-06 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501551+0000", 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5109999999999997e-06 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515586+0000", 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014035334999999999 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515689+0000", 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010244200000000001 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515775+0000", 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6150999999999996e-05 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11679 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.615 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.036569184000001, 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014462922, 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11679 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:36.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501586+0000", 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501588+0000", 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3760000000000002e-06 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501594+0000", 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9950000000000002e-06 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501598+0000", 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7239999999999998e-06 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515840+0000", 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014242546999999999 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515949+0000", 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108591 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516048+0000", 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9123000000000006e-05 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11686 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.036552612000001, 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014648357000000001, 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.620 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11686 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501604+0000", 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.846e-06 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501609+0000", 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3630000000000002e-06 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501613+0000", 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3749999999999999e-06 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516077+0000", 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014464351 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516173+0000", 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5742000000000006e-05 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516250+0000", 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7312999999999997e-05 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11689 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.624 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.036538048000001, 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014770922000000001, 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11689 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:36.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501617+0000", 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501618+0000", 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6780000000000001e-06 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501624+0000", 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3750000000000002e-06 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501627+0000", 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5149999999999998e-06 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516279+0000", 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014651928999999999 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516336+0000", 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6724000000000001e-05 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516387+0000", 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1334e-05 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11778 3.1 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.019203919999999, 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022495198000000001, 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:36.629 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11778 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:36.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518952+0000", 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.711e-06 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522261+0000", 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033087429999999998 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:36.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522296+0000", 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5335e-05 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522306+0000", 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0074999999999999e-05 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522379+0000", 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2554999999999999e-05 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522572+0000", 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019323100000000001 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.523266+0000", 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069379000000000001 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.534086+0000", 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010820112999999999 2024-09-16T13:17:36.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541312+0000", 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072264440000000003 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541394+0000", 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1710000000000002e-05 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541446+0000", 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1490999999999999e-05 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.634 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:36.635 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:36.635 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:36.635 INFO:teuthology.orchestra.run.smithi098.stdout:2765 writesame 0xd0aa17c thru 0xd0ab844 data_size 0x1(0x16c9 bytes) 2024-09-16T13:17:36.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:36.637 INFO:teuthology.orchestra.run.smithi098.stdout:2766 trunc from 0xd0ab845 to 0x4ad9696 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:2770 punch from 0x12bee20 to 0x12cdff2, (0xf1d2 bytes) 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:2771 punch from 0x46c2d91 to 0x46c903c, (0x62ab bytes) 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:2772 read 0x2e714cc thru 0x2e7d1c4 (0xbcf9 bytes) 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:2773 trunc from 0x4ad9696 to 0x378b90f 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:2774 punch from 0x1e7db5a to 0x1e7e791, (0xc37 bytes) 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:2779 write 0xb36603 thru 0xb43743 (0xd141 bytes) 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:2781 punch from 0x112a15f to 0x112a77e, (0x61f bytes) 2024-09-16T13:17:36.638 INFO:teuthology.orchestra.run.smithi098.stdout:2782 write 0xabeef40 thru 0xabf9b2b (0xabec bytes) 2024-09-16T13:17:36.639 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:36.631+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:36.640 INFO:teuthology.orchestra.run.smithi098.stdout:2783 read 0x101b61 thru 0x10f4dc (0xd97c bytes) 2024-09-16T13:17:36.642 INFO:teuthology.orchestra.run.smithi098.stdout:2784 read 0x8b3bee thru 0x8bf9e7 (0xbdfa bytes) 2024-09-16T13:17:36.642 INFO:teuthology.orchestra.run.smithi098.stdout:2785 write 0x94f3a74 thru 0x94f67ba (0x2d47 bytes) 2024-09-16T13:17:36.648 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.648 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.668 INFO:teuthology.orchestra.run.smithi098.stdout:2792 trunc from 0xabf9b2c to 0x1e9573f 2024-09-16T13:17:36.693 INFO:teuthology.orchestra.run.smithi098.stdout:2793 read 0x598789 thru 0x5a825a (0xfad2 bytes) 2024-09-16T13:17:36.694 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.694 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.694 INFO:teuthology.orchestra.run.smithi098.stdout:2798 punch from 0xf3be24 to 0xf46fbe, (0xb19a bytes) 2024-09-16T13:17:36.713 INFO:teuthology.orchestra.run.smithi098.stdout:2799 write 0x4ee48c0 thru 0x4ee8c5e (0x439f bytes) 2024-09-16T13:17:36.739 INFO:teuthology.orchestra.run.smithi098.stdout:2801 writesame 0x73ab33b thru 0x73b516d data_size 0x1(0x9e33 bytes) 2024-09-16T13:17:36.739 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:36.739 INFO:teuthology.orchestra.run.smithi098.stdout:2805 punch from 0x5022082 to 0x5026faf, (0x4f2d bytes) 2024-09-16T13:17:36.739 INFO:teuthology.orchestra.run.smithi098.stdout:2806 clone 16 order 21 su 262144 sc 10 2024-09-16T13:17:36.740 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:36.731+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:36.751 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:36.751 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:36.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:36.841 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:36.835+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:36.862 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:36.862 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:36.862 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-16T13:17:36.862 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:17:36.862 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:17:36.862 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-16T13:17:36.910 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:36.910 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:36.910 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:36.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:36.942 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:36.935+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:36.956 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:36.956 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T13:17:36.956 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:37.022 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.022 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6966 3.0 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.461429321000001, 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095509229999999994, 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6966 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554532+0000", 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2000000000000001e-07 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554534+0000", 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9850000000000001e-06 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554537+0000", 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.664e-06 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554541+0000", 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2909999999999999e-06 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557170+0000", 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026295310000000001 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557345+0000", 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017473200000000001 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557618+0000", 2024-09-16T13:17:37.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027282800000000002 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563504+0000", 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058862649999999999 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563923+0000", 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000419004 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564003+0000", 2024-09-16T13:17:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0072000000000002e-05 2024-09-16T13:17:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564082+0000", 2024-09-16T13:17:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9231000000000001e-05 2024-09-16T13:17:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.028 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11223 3.8 e40/23 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head v 40'368, mlcod=39'365)", 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.549258237, 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01312405, 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466704+0000", 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.792e-06 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471144+0000", 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044394830000000001 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471180+0000", 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5884999999999999e-05 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471194+0000", 2024-09-16T13:17:37.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4258e-05 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471273+0000", 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9159999999999994e-05 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471305+0000", 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2236000000000001e-05 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479753+0000", 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0084474049999999998 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479826+0000", 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3831000000000005e-05 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11776 3.1a e40/23 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head v 40'72, mlcod=40'70)", 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:37.033 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.488361439999998, 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595333, 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527601+0000", 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.522e-06 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536775+0000", 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091746569999999993 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536818+0000", 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2435000000000003e-05 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536828+0000", 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0624e-05 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536931+0000", 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010280000000000001 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536961+0000", 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0060999999999999e-05 2024-09-16T13:17:37.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544146+0000", 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071846339999999996 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544195+0000", 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8600000000000002e-05 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13719 3.19 3:99fbe691:::rbd_data.112df1cc59bd.000000000000000c:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.735191868999999, 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011713708999999999, 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13719 2024-09-16T13:17:37.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280771+0000", 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9240000000000001e-06 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280782+0000", 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1642e-05 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280806+0000", 2024-09-16T13:17:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4119000000000001e-05 2024-09-16T13:17:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280815+0000", 2024-09-16T13:17:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7150000000000004e-06 2024-09-16T13:17:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012396900000000001 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281142+0000", 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000203298 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281540+0000", 2024-09-16T13:17:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039763500000000002 2024-09-16T13:17:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.286598+0000", 2024-09-16T13:17:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050579090000000002 2024-09-16T13:17:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292380+0000", 2024-09-16T13:17:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057823390000000001 2024-09-16T13:17:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292433+0000", 2024-09-16T13:17:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2612999999999999e-05 2024-09-16T13:17:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292482+0000", 2024-09-16T13:17:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9546000000000003e-05 2024-09-16T13:17:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.045 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13720 3.19 3:99acede8:::rbd_data.112df1cc59bd.000000000000000d:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.735128677000001, 2024-09-16T13:17:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011830879000000001, 2024-09-16T13:17:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.046 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13720 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:37.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280836+0000", 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.703e-06 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280844+0000", 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280849+0000", 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.254e-06 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282739+0000", 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001889672 2024-09-16T13:17:37.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282875+0000", 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013641 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.283217+0000", 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00034147099999999998 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.288528+0000", 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053110140000000002 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292558+0000", 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040304429999999999 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292620+0000", 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1588999999999996e-05 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292663+0000", 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2922e-05 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13726 3.19 3:9e45447e:::rbd_data.112df1cc59bd.0000000000000034:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.735027964, 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012171434, 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.052 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13726 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280934+0000", 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9299999999999996e-07 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280950+0000", 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1141000000000001e-05 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280955+0000", 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5560000000000001e-06 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284230+0000", 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032752789999999999 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284286+0000", 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6456999999999998e-05 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284430+0000", 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014382899999999999 2024-09-16T13:17:37.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.291179+0000", 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067490099999999997 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293014+0000", 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018347719999999999 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293069+0000", 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5325999999999998e-05 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293104+0000", 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4892999999999999e-05 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14647 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 3801088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.380830027, 2024-09-16T13:17:37.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0094151970000000001, 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14647 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635132+0000", 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4300000000000001e-07 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635136+0000", 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3660000000000002e-06 2024-09-16T13:17:37.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635150+0000", 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4491999999999999e-05 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635159+0000", 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2630000000000003e-06 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644405+0000", 2024-09-16T13:17:37.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0092462220000000001 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644476+0000", 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0958999999999996e-05 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644546+0000", 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9951999999999997e-05 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.062 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14648 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4325376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.380792339999999, 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096264239999999997, 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14648 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635169+0000", 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.071e-06 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635173+0000", 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635188+0000", 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4389000000000001e-05 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635198+0000", 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0566e-05 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644579+0000", 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093805399999999997 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644670+0000", 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1240000000000001e-05 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644795+0000", 2024-09-16T13:17:37.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012454 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14662 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1703936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.380546774000001, 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096453119999999996, 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14662 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635415+0000", 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5e-06 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635421+0000", 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730000000000001e-06 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635436+0000", 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4909000000000001e-05 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635450+0000", 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3183999999999999e-05 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644833+0000", 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093835959999999993 2024-09-16T13:17:37.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644914+0000", 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0624000000000006e-05 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645059+0000", 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014542600000000001 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14663 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2228224~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.380501612, 2024-09-16T13:17:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098295339999999991, 2024-09-16T13:17:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14663 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635460+0000", 2024-09-16T13:17:37.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1500000000000004e-07 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635463+0000", 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.478e-06 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635472+0000", 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4230000000000005e-06 2024-09-16T13:17:37.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635498+0000", 2024-09-16T13:17:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6543e-05 2024-09-16T13:17:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645115+0000", 2024-09-16T13:17:37.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096171710000000007 2024-09-16T13:17:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645193+0000", 2024-09-16T13:17:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7245999999999999e-05 2024-09-16T13:17:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645289+0000", 2024-09-16T13:17:37.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5958000000000005e-05 2024-09-16T13:17:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14664 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.380455122000001, 2024-09-16T13:17:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010033297, 2024-09-16T13:17:37.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14664 2024-09-16T13:17:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635506+0000", 2024-09-16T13:17:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5099999999999998e-07 2024-09-16T13:17:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635511+0000", 2024-09-16T13:17:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7820000000000001e-06 2024-09-16T13:17:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635520+0000", 2024-09-16T13:17:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5969999999999992e-06 2024-09-16T13:17:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635529+0000", 2024-09-16T13:17:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3249999999999997e-06 2024-09-16T13:17:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645337+0000", 2024-09-16T13:17:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098082589999999997 2024-09-16T13:17:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645427+0000", 2024-09-16T13:17:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9922999999999999e-05 2024-09-16T13:17:37.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645539+0000", 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011156 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14665 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.380425894, 2024-09-16T13:17:37.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010434005999999999, 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14665 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:37.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635538+0000", 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8329999999999998e-06 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635546+0000", 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0760000000000003e-06 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635551+0000", 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6160000000000004e-06 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645677+0000", 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010126546 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645821+0000", 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014406799999999999 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645969+0000", 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014730900000000001 2024-09-16T13:17:37.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14678 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 0~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.380166450000001, 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012023611999999999, 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.089 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14678 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:37.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635797+0000", 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7039999999999999e-06 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635810+0000", 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3343000000000001e-05 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635845+0000", 2024-09-16T13:17:37.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4751e-05 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635863+0000", 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8210999999999999e-05 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646005+0000", 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010141766999999999 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646070+0000", 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5324e-05 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647818+0000", 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017475119999999999 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14679 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1835008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.093 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.380078744, 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012069985, 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14679 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635883+0000", 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3239999999999999e-06 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635889+0000", 2024-09-16T13:17:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:17:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635905+0000", 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6022999999999999e-05 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635911+0000", 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3740000000000003e-06 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647849+0000", 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011937906999999999 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647895+0000", 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5507999999999999e-05 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647952+0000", 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7132000000000001e-05 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14680 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2359296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.380042462, 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012172090999999999, 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14680 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.099 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635919+0000", 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8299999999999995e-07 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635922+0000", 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.027e-06 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635931+0000", 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6230000000000003e-06 2024-09-16T13:17:37.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635936+0000", 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0069999999999997e-06 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647981+0000", 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012045273 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648033+0000", 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648090+0000", 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7154000000000002e-05 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14681 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4718592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.380019465, 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012312528, 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14681 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635942+0000", 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7599999999999996e-07 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635959+0000", 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6954999999999999e-05 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635968+0000", 2024-09-16T13:17:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8929999999999994e-06 2024-09-16T13:17:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635973+0000", 2024-09-16T13:17:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0170000000000004e-06 2024-09-16T13:17:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648120+0000", 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012146847000000001 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648170+0000", 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0368000000000003e-05 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648254+0000", 2024-09-16T13:17:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3671999999999994e-05 2024-09-16T13:17:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14694 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.379982391, 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012792843999999999, 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14694 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635979+0000", 2024-09-16T13:17:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0699999999999996e-07 2024-09-16T13:17:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635982+0000", 2024-09-16T13:17:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8200000000000001e-06 2024-09-16T13:17:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635990+0000", 2024-09-16T13:17:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0239999999999997e-06 2024-09-16T13:17:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635995+0000", 2024-09-16T13:17:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648296+0000", 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0123009 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648397+0000", 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101042 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648771+0000", 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037435099999999998 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14695 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1966080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.379960372999999, 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012992136, 2024-09-16T13:17:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14695 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636001+0000", 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0999999999999998e-07 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636004+0000", 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5959999999999998e-06 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636012+0000", 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636016+0000", 2024-09-16T13:17:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9269999999999996e-06 2024-09-16T13:17:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648804+0000", 2024-09-16T13:17:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012787543 2024-09-16T13:17:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648873+0000", 2024-09-16T13:17:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9263000000000004e-05 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648992+0000", 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119089 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14696 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2490368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.379938682000001, 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013132612, 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14696 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636023+0000", 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7300000000000005e-07 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636025+0000", 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.407e-06 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636033+0000", 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0129999999999993e-06 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636042+0000", 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1540000000000008e-06 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649027+0000", 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012985096 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649089+0000", 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1641000000000002e-05 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649155+0000", 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5528e-05 2024-09-16T13:17:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14697 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.379912781, 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013321177999999999, 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14697 2024-09-16T13:17:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636049+0000", 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636051+0000", 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.57e-06 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636059+0000", 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9440000000000005e-06 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636067+0000", 2024-09-16T13:17:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0460000000000006e-06 2024-09-16T13:17:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649182+0000", 2024-09-16T13:17:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01311522 2024-09-16T13:17:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649245+0000", 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2930999999999998e-05 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649369+0000", 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123681 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.129 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:37.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:37.131 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:37.127+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:37.232 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:37.224+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:37.238 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:37.238 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:37.238 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:37.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:37.332 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:37.327+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:37.338 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:37.338 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:37.338 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:37.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:37.434 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:37.431+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1264 3.4 e27/23 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head v 27'4, mlcod=26'2)", 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 37.946808838999999, 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026410209999999999, 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501250+0000", 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2410000000000001e-06 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501259+0000", 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501277+0000", 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8683999999999999e-05 2024-09-16T13:17:37.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501288+0000", 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.065e-05 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501562+0000", 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027421200000000002 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501618+0000", 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6045000000000003e-05 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503841+0000", 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022230829999999998 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503890+0000", 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8526000000000003e-05 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.440665150999997, 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:17:37.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.439879652000002, 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.467 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.438712729999999, 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:17:37.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.435838099000001, 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:17:37.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.432824359000001, 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:17:37.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.407620156, 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.481 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:17:37.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.641471267, 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:37.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:17:37.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5115 3.4 e34/23 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head v 34'14, mlcod=31'12)", 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.656423160999999, 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029400680000000001, 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791635+0000", 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4200000000000005e-07 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791643+0000", 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.157e-06 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791660+0000", 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6246e-05 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791666+0000", 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791743+0000", 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7000000000000001e-05 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791756+0000", 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2729e-05 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794523+0000", 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027675949999999999 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794575+0000", 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1082000000000003e-05 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.494 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 25.389368963999999, 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:17:37.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 25.388759524000001, 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.499 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6963 3.4 e37/23 3:2713f365:::rbd_data.112d6fb8d387.0000000000000133:head v 37'31, mlcod=37'29)", 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.892195374, 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027598190000000002, 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555864+0000", 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555882+0000", 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7816e-05 2024-09-16T13:17:37.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555909+0000", 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7838999999999999e-05 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555923+0000", 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3577000000000001e-05 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556083+0000", 2024-09-16T13:17:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015987299999999999 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556132+0000", 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8655000000000001e-05 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558539+0000", 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024076969999999999 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558622+0000", 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2777999999999999e-05 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.891475710000002, 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.508 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.890374894000001, 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:17:37.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.890235254, 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:17:37.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:17:37.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:37.521 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.889734724, 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:17:37.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.525 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:10843 3.0 e40/23 3:039a6044:::rbd_data.112ddef65b5a.0000000000000050:head v 40'68, mlcod=39'65)", 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.242323+0000", 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.205734306, 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025010879999999998, 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242323+0000", 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242323+0000", 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242324+0000", 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242332+0000", 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5529999999999993e-06 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242354+0000", 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1336999999999999e-05 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242362+0000", 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9859999999999994e-06 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242409+0000", 2024-09-16T13:17:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7416000000000001e-05 2024-09-16T13:17:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242431+0000", 2024-09-16T13:17:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2291000000000001e-05 2024-09-16T13:17:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.244771+0000", 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023402280000000002 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.244824+0000", 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.894565338, 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:37.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:17:37.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.6381514969999991, 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:17:37.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:17:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:17:37.540 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.0255883899999994, 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:17:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:17:37.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.545 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.545 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.545 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:37.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:37.547 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:37.540+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:37.647 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:37.639+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:37.656 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:37.656 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:37.656 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:37.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:37.748 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:37.740+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:37.763 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:37.764 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:37.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:37.807 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #14, image image_client.0-clone14 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent15 2024-09-16T13:17:37.849 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:37.839+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:37.873 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5101 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 30.090329231999998, 2024-09-16T13:17:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595248, 2024-09-16T13:17:37.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5101 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777333+0000", 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1949999999999996e-06 2024-09-16T13:17:37.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777342+0000", 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6950000000000006e-06 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777348+0000", 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785241+0000", 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078929329999999995 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785391+0000", 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150113 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785969+0000", 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057822899999999998 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789745+0000", 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037756529999999999 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793823+0000", 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040786299999999998 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793880+0000", 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7157999999999997e-05 2024-09-16T13:17:37.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793924+0000", 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3407e-05 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5102 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 30.090281940000001, 2024-09-16T13:17:37.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016663710000000002, 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5102 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777377+0000", 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4600000000000003e-07 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777381+0000", 2024-09-16T13:17:37.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5290000000000002e-06 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777393+0000", 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2001000000000001e-05 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777401+0000", 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3869999999999997e-06 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786753+0000", 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093523649999999996 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786860+0000", 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106921 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787286+0000", 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042651800000000003 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793815+0000", 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065290579999999999 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793932+0000", 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116688 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794002+0000", 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0313000000000003e-05 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794040+0000", 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7083999999999999e-05 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5103 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 30.090221054000001, 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017505544000000001, 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5103 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777438+0000", 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.78e-07 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777442+0000", 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5950000000000001e-06 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777453+0000", 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0372e-05 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777458+0000", 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6559999999999997e-06 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788034+0000", 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010575753 2024-09-16T13:17:37.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788111+0000", 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7038000000000003e-05 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788463+0000", 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000352028 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794263+0000", 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005800342 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794837+0000", 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057334300000000003 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794897+0000", 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0690000000000003e-05 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794942+0000", 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4848999999999997e-05 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5104 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 30.090176646, 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017694142999999999, 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.891 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5104 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777482+0000", 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777487+0000", 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6639999999999997e-06 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777496+0000", 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2890000000000003e-06 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777501+0000", 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2730000000000002e-06 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788951+0000", 2024-09-16T13:17:37.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011449429000000001 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789007+0000", 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5643999999999998e-05 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789205+0000", 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019820800000000001 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794268+0000", 2024-09-16T13:17:37.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050636850000000001 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795094+0000", 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082569700000000002 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795139+0000", 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5055000000000001e-05 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795175+0000", 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6149000000000002e-05 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5105 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 30.090114305, 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017729482000000001, 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.897 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5105 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777544+0000", 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1100000000000004e-07 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777549+0000", 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777561+0000", 2024-09-16T13:17:37.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1524e-05 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777566+0000", 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.558e-06 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789781+0000", 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012214981 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789870+0000", 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9171000000000004e-05 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790313+0000", 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044255299999999999 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794273+0000", 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039596129999999998 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795213+0000", 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094086099999999998 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795239+0000", 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5324e-05 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795273+0000", 2024-09-16T13:17:37.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4198e-05 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5106 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 30.090069160999999, 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018621867, 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.903 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5106 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:37.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777590+0000", 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777594+0000", 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6750000000000001e-06 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777604+0000", 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0076e-05 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777610+0000", 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7529999999999998e-06 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790893+0000", 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013282845 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790999+0000", 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106289 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791493+0000", 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049357800000000005 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794662+0000", 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031691409999999999 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796146+0000", 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00148375 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796176+0000", 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0349e-05 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796211+0000", 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4530000000000003e-05 2024-09-16T13:17:37.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5107 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 30.090037899999999, 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018745067000000001, 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.909 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5107 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777621+0000", 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-07 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777624+0000", 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7950000000000001e-06 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777633+0000", 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5979999999999997e-06 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777638+0000", 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.989e-06 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792024+0000", 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014385692 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792104+0000", 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0567000000000003e-05 2024-09-16T13:17:37.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792340+0000", 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023624200000000001 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794667+0000", 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023269990000000002 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796304+0000", 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001636981 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796331+0000", 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6492999999999999e-05 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796365+0000", 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4102e-05 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5108 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 30.089984071, 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019090617000000001, 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5108 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777677+0000", 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7879999999999998e-06 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777686+0000", 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7679999999999997e-06 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777689+0000", 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.123e-06 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792751+0000", 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01506158 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792855+0000", 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010456 2024-09-16T13:17:37.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793305+0000", 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045006100000000003 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794671+0000", 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001366304 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796393+0000", 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001721272 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796733+0000", 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000340611 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796764+0000", 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.099e-05 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5208 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.98822462, 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014882177999999999, 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5208 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.921 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879434+0000", 2024-09-16T13:17:37.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.322e-06 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879443+0000", 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879459+0000", 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6133999999999998e-05 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879467+0000", 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2400000000000007e-06 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888339+0000", 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088721619999999994 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888461+0000", 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121341 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888819+0000", 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035782799999999998 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891271+0000", 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002452872 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894224+0000", 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029522910000000001 2024-09-16T13:17:37.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894275+0000", 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1647999999999997e-05 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894315+0000", 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9993999999999997e-05 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5210 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.987669432000001, 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01466423, 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5210 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879989+0000", 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8800000000000001e-07 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879995+0000", 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880017+0000", 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2101000000000001e-05 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880024+0000", 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9240000000000001e-06 2024-09-16T13:17:37.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890250+0000", 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010226012 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890344+0000", 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.446e-05 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890770+0000", 2024-09-16T13:17:37.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042575099999999998 2024-09-16T13:17:37.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894344+0000", 2024-09-16T13:17:37.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003574163 2024-09-16T13:17:37.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894549+0000", 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020477599999999999 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894606+0000", 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6891000000000001e-05 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894653+0000", 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6867999999999999e-05 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5211 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.987625369, 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014800604, 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5211 2024-09-16T13:17:37.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880033+0000", 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.06e-07 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880039+0000", 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880052+0000", 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-05 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880059+0000", 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3090000000000001e-06 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891316+0000", 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011257118999999999 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891374+0000", 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8020999999999999e-05 2024-09-16T13:17:37.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891577+0000", 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020371699999999999 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894348+0000", 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027708300000000002 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894726+0000", 2024-09-16T13:17:37.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037819399999999999 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894784+0000", 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7667999999999997e-05 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894833+0000", 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8936e-05 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5212 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.987269994999998, 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014645704000000001, 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5212 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.939 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880389+0000", 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1680000000000001e-06 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880395+0000", 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880408+0000", 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.308e-05 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880414+0000", 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.105e-06 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892037+0000", 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011622402 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892116+0000", 2024-09-16T13:17:37.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9939999999999997e-05 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892597+0000", 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048076500000000002 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894492+0000", 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001895246 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894921+0000", 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000428114 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894985+0000", 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4777999999999995e-05 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895034+0000", 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8133999999999997e-05 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5213 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.987136844999998, 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014632608, 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5213 2024-09-16T13:17:37.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880522+0000", 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2600000000000001e-07 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880527+0000", 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880538+0000", 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917999999999999e-05 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880543+0000", 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.361e-06 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893069+0000", 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012525497999999999 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893152+0000", 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2805000000000004e-05 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893635+0000", 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048315100000000002 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894852+0000", 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001216806 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895075+0000", 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022300299999999999 2024-09-16T13:17:37.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895122+0000", 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7077999999999999e-05 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895154+0000", 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1919999999999999e-05 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5214 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.986982193999999, 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017785977000000001, 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5214 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880681+0000", 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880689+0000", 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3610000000000001e-06 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880694+0000", 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5809999999999998e-06 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895186+0000", 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014492430000000001 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895311+0000", 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012518300000000001 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895760+0000", 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044864900000000002 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898337+0000", 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025765229999999998 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898354+0000", 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7538999999999999e-05 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898406+0000", 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2054e-05 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898462+0000", 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5404e-05 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5215 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.98671736, 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018191319000000001, 2024-09-16T13:17:37.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5215 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880941+0000", 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880946+0000", 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3710000000000002e-06 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880955+0000", 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1230000000000006e-06 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880959+0000", 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896782+0000", 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015822716000000001 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896876+0000", 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3667999999999998e-05 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.897289+0000", 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041357399999999999 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898474+0000", 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011847209999999999 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899036+0000", 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056215699999999998 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899090+0000", 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3662999999999998e-05 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899132+0000", 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1814000000000003e-05 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5282 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.918045245999998, 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014924988, 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5282 2024-09-16T13:17:37.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:37.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.53e-07 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949635+0000", 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2422999999999999e-05 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949641+0000", 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.896e-06 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949644+0000", 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5270000000000001e-06 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959656+0000", 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010012378000000001 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959747+0000", 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1273000000000005e-05 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960045+0000", 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029733 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963196+0000", 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031512620000000002 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964453+0000", 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001257357 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964502+0000", 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9044e-05 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964538+0000", 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5244999999999999e-05 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5283 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:37.967 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.918010091999999, 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015065762, 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5283 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949650+0000", 2024-09-16T13:17:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.736e-06 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949653+0000", 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2969999999999999e-06 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949655+0000", 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1890000000000001e-06 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960643+0000", 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010988015 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960732+0000", 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9097000000000004e-05 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961195+0000", 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000462292 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963208+0000", 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002013641 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964585+0000", 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013765769999999999 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964669+0000", 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3863000000000007e-05 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964713+0000", 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4706999999999997e-05 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5284 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.918000223, 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015152802, 2024-09-16T13:17:37.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5284 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5899999999999998e-07 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949659+0000", 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.601e-06 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949672+0000", 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2364999999999999e-05 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949674+0000", 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961869+0000", 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012194903999999999 2024-09-16T13:17:37.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961971+0000", 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102154 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.962502+0000", 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00053037500000000003 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964751+0000", 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022487800000000001 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964755+0000", 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964789+0000", 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4054000000000003e-05 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964810+0000", 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0922000000000001e-05 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5285 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.917981305000001, 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015553022, 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5285 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949677+0000", 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6899999999999999e-07 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949679+0000", 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0090000000000002e-06 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949682+0000", 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2550000000000002e-06 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949685+0000", 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.621e-06 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963409+0000", 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013724485999999999 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963497+0000", 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7560999999999997e-05 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963933+0000", 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043592700000000002 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964820+0000", 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088772299999999998 2024-09-16T13:17:37.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965165+0000", 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000344426 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965201+0000", 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5967000000000003e-05 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965230+0000", 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:17:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11223 3.8 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 33927~20122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.406808975000001, 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019333487, 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:37.985 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11223 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:37.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460851+0000", 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5730000000000002e-06 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464824+0000", 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039725730000000001 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464861+0000", 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7233000000000001e-05 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:37.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464872+0000", 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209000000000001e-05 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464987+0000", 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011423 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465188+0000", 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020091600000000001 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:37.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465929+0000", 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00074102 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.475697+0000", 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009768367 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480068+0000", 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043713290000000002 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:37.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480143+0000", 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5000999999999996e-05 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480182+0000", 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9035999999999997e-05 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:37.991 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:37.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:37.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:37.994 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:37.984+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:38.009 INFO:teuthology.orchestra.run.smithi098.stdout:2807 read 0x3d3ab0a thru 0x3d3b37a (0x871 bytes) 2024-09-16T13:17:38.032 INFO:teuthology.orchestra.run.smithi098.stdout:2809 write 0xaa0760d thru 0xaa12f58 (0xb94c bytes) 2024-09-16T13:17:38.056 INFO:teuthology.orchestra.run.smithi098.stdout:2810 write 0xe13ec73 thru 0xe1419e6 (0x2d74 bytes) 2024-09-16T13:17:38.063 INFO:teuthology.orchestra.run.smithi098.stdout:2811 write 0xa675c8b thru 0xa67e02a (0x83a0 bytes) 2024-09-16T13:17:38.069 INFO:teuthology.orchestra.run.smithi098.stdout:2812 write 0xb39de0e thru 0xb3a58f1 (0x7ae4 bytes) 2024-09-16T13:17:38.075 INFO:teuthology.orchestra.run.smithi098.stdout:2814 punch from 0xb083ae8 to 0xb08b47a, (0x7992 bytes) 2024-09-16T13:17:38.075 INFO:teuthology.orchestra.run.smithi098.stdout:2815 write 0x74e7dce thru 0x74eaab0 (0x2ce3 bytes) 2024-09-16T13:17:38.081 INFO:teuthology.orchestra.run.smithi098.stdout:2816 write 0xbd52b2f thru 0xbd5fbce (0xd0a0 bytes) 2024-09-16T13:17:38.088 INFO:teuthology.orchestra.run.smithi098.stdout:2817 writesame 0x8a4dc63 thru 0x8a5ba61 data_size 0x1(0xddff bytes) 2024-09-16T13:17:38.094 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:38.084+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:38.110 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:38.110 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:38.110 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:38.110 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:38.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:38.146 INFO:teuthology.orchestra.run.smithi098.stdout:2818 trunc from 0xe1419e7 to 0x8a22359 2024-09-16T13:17:38.169 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:38.172 INFO:teuthology.orchestra.run.smithi098.stdout:2819 read 0x75bc232 thru 0x75c1aae (0x587d bytes) 2024-09-16T13:17:38.173 INFO:teuthology.orchestra.run.smithi098.stdout:2820 punch from 0x308f91d to 0x309b744, (0xbe27 bytes) 2024-09-16T13:17:38.173 INFO:teuthology.orchestra.run.smithi098.stdout:2822 read 0x2ec4dc5 thru 0x2ecd0fb (0x8337 bytes) 2024-09-16T13:17:38.192 INFO:teuthology.orchestra.run.smithi098.stdout:2823 write 0xd7c850e thru 0xd7d5fa6 (0xda99 bytes) 2024-09-16T13:17:38.194 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:38.184+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:38.195 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:38.198 INFO:teuthology.orchestra.run.smithi098.stdout:2824 writesame 0x8b1c1cf thru 0x8b21bec data_size 0xf(0x5a1e bytes) 2024-09-16T13:17:38.206 INFO:teuthology.orchestra.run.smithi098.stdout:2825 read 0x3272a62 thru 0x327d33c (0xa8db bytes) 2024-09-16T13:17:38.218 INFO:teuthology.orchestra.run.smithi098.stdout:2826 write 0xe31e5ce thru 0xe32a06d (0xbaa0 bytes) 2024-09-16T13:17:38.220 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:38.220 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:38.221 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:38.221 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:38.221 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:38.221 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:38.224 INFO:teuthology.orchestra.run.smithi098.stdout:2827 write 0x645f2a3 thru 0x6469aed (0xa84b bytes) 2024-09-16T13:17:38.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:38.231 INFO:teuthology.orchestra.run.smithi098.stdout:2828 punch from 0x4441cea to 0x4448381, (0x6697 bytes) 2024-09-16T13:17:38.231 INFO:teuthology.orchestra.run.smithi098.stdout:2829 writesame 0xc52f46e thru 0xc5370cf data_size 0x3d(0x7c62 bytes) 2024-09-16T13:17:38.237 INFO:teuthology.orchestra.run.smithi098.stdout:2830 punch from 0xa810814 to 0xa810c82, (0x46e bytes) 2024-09-16T13:17:38.237 INFO:teuthology.orchestra.run.smithi098.stdout:2831 read 0x69aabfa thru 0x69b8eea (0xe2f1 bytes) 2024-09-16T13:17:38.237 INFO:teuthology.orchestra.run.smithi098.stdout:2832 read 0x47e8b36 thru 0x47ed4aa (0x4975 bytes) 2024-09-16T13:17:38.237 INFO:teuthology.orchestra.run.smithi098.stdout:2833 writesame 0xd56fe21 thru 0xd57d8e0 data_size 0x7d(0xdac0 bytes) 2024-09-16T13:17:38.246 INFO:teuthology.orchestra.run.smithi098.stdout:2834 trunc from 0xe32a06e to 0xb885ee5 2024-09-16T13:17:38.246 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:38.268 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:38.282 INFO:teuthology.orchestra.run.smithi098.stdout:2835 write 0xe2870ef thru 0xe2945f0 (0xd502 bytes) 2024-09-16T13:17:38.289 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:38.295 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:38.287+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:38.297 INFO:teuthology.orchestra.run.smithi098.stdout:2836 writesame 0xe32480d thru 0xe3325dd data_size 0x29(0xddd1 bytes) 2024-09-16T13:17:38.298 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.298 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.298 INFO:teuthology.orchestra.run.smithi098.stdout:2841 write 0x47de6f0 thru 0x47e6161 (0x7a72 bytes) 2024-09-16T13:17:38.304 INFO:teuthology.orchestra.run.smithi098.stdout:2842 read 0xd023270 thru 0xd0287e3 (0x5574 bytes) 2024-09-16T13:17:38.304 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.304 INFO:teuthology.orchestra.run.smithi098.stdout:2844 punch from 0x804cd82 to 0x805960c, (0xc88a bytes) 2024-09-16T13:17:38.308 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:38.322 INFO:teuthology.orchestra.run.smithi098.stdout:2845 trunc from 0xe3325de to 0x395c11f 2024-09-16T13:17:38.323 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:38.323 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:38.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:38.362 INFO:teuthology.orchestra.run.smithi098.stdout:2846 trunc from 0x395c11f to 0x3278662 2024-09-16T13:17:38.384 INFO:teuthology.orchestra.run.smithi098.stdout:2847 trunc from 0x3278662 to 0x69d960b 2024-09-16T13:17:38.396 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:38.387+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:38.416 INFO:teuthology.orchestra.run.smithi098.stdout:2849 write 0xad0edeb thru 0xad13abe (0x4cd4 bytes) 2024-09-16T13:17:38.423 INFO:teuthology.orchestra.run.smithi098.stdout:2852 read 0x18780a7 thru 0x18873a0 (0xf2fa bytes) 2024-09-16T13:17:38.424 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.424 INFO:teuthology.orchestra.run.smithi098.stdout:2854 writesame 0x6d58b75 thru 0x6d5dd0a data_size 0x6(0x5196 bytes) 2024-09-16T13:17:38.431 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:38.431 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:38.435 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.435 INFO:teuthology.orchestra.run.smithi098.stdout:2857 write 0x562cd7b thru 0x5633c65 (0x6eeb bytes) 2024-09-16T13:17:38.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:38.442 INFO:teuthology.orchestra.run.smithi098.stdout:2858 writesame 0x767ef57 thru 0x768aae8 data_size 0x97(0xbb92 bytes) 2024-09-16T13:17:38.455 INFO:teuthology.orchestra.run.smithi098.stdout:2859 read 0x596f1c1 thru 0x59728ea (0x372a bytes) 2024-09-16T13:17:38.455 INFO:teuthology.orchestra.run.smithi098.stdout:2860 read 0xac0fbde thru 0xac1824b (0x866e bytes) 2024-09-16T13:17:38.455 INFO:teuthology.orchestra.run.smithi098.stdout:2861 writesame 0x24de745 thru 0x24e2f89 data_size 0x7(0x4845 bytes) 2024-09-16T13:17:38.466 INFO:teuthology.orchestra.run.smithi098.stdout:2862 writesame 0x7b1a607 thru 0x7b24bf9 data_size 0x77(0xa5f3 bytes) 2024-09-16T13:17:38.474 INFO:teuthology.orchestra.run.smithi098.stdout:2863 writesame 0x8eb88ad thru 0x8ec0c0d data_size 0x25(0x8361 bytes) 2024-09-16T13:17:38.482 INFO:teuthology.orchestra.run.smithi098.stdout:2864 write 0x6c932e6 thru 0x6c954cc (0x21e7 bytes) 2024-09-16T13:17:38.485 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.485 INFO:teuthology.orchestra.run.smithi098.stdout:2866 read 0x2bbdb41 thru 0x2bc8830 (0xacf0 bytes) 2024-09-16T13:17:38.486 INFO:teuthology.orchestra.run.smithi098.stdout:2867 punch from 0x54fea27 to 0x550d460, (0xea39 bytes) 2024-09-16T13:17:38.497 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:38.487+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:38.504 INFO:teuthology.orchestra.run.smithi098.stdout:2868 writesame 0xcb8e7c6 thru 0xcb919fe data_size 0x1(0x3239 bytes) 2024-09-16T13:17:38.504 INFO:teuthology.orchestra.run.smithi098.stdout:2869 punch from 0x807dc40 to 0x80871fb, (0x95bb bytes) 2024-09-16T13:17:38.505 INFO:teuthology.orchestra.run.smithi098.stdout:2872 write 0x6575466 thru 0x6584858 (0xf3f3 bytes) 2024-09-16T13:17:38.528 INFO:teuthology.orchestra.run.smithi098.stdout:2873 trunc from 0xcb919ff to 0x6336f9a 2024-09-16T13:17:38.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:17:38.563 INFO:teuthology.orchestra.run.smithi098.stdout:2874 write 0x7b7c5dc thru 0x7b893d2 (0xcdf7 bytes) 2024-09-16T13:17:38.571 INFO:teuthology.orchestra.run.smithi098.stdout:2876 write 0x396b971 thru 0x3972b49 (0x71d9 bytes) 2024-09-16T13:17:38.574 INFO:teuthology.orchestra.run.smithi098.stdout:2877 read 0x3d5755 thru 0x3e02b4 (0xab60 bytes) 2024-09-16T13:17:38.575 INFO:teuthology.orchestra.run.smithi098.stdout:2879 read 0x3d8589b thru 0x3d8d0d8 (0x783e bytes) 2024-09-16T13:17:38.575 INFO:teuthology.orchestra.run.smithi098.stdout:2880 read 0x5664594 thru 0x566f5f8 (0xb065 bytes) 2024-09-16T13:17:38.575 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.575 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.595 INFO:teuthology.orchestra.run.smithi098.stdout:2883 writesame 0xb28fa2f thru 0xb29200d data_size 0x7(0x25df bytes) 2024-09-16T13:17:38.595 INFO:teuthology.orchestra.run.smithi098.stdout:2884 read 0x275c012 thru 0x275cbc0 (0xbaf bytes) 2024-09-16T13:17:38.597 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:38.588+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:38.602 INFO:teuthology.orchestra.run.smithi098.stdout:2886 trunc from 0xb29200e to 0x755e899 2024-09-16T13:17:38.604 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:38.604 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:38.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:38.648 INFO:teuthology.orchestra.run.smithi098.stdout:2887 writesame 0xdb89a4f thru 0xdb8c546 data_size 0x19(0x2af8 bytes) 2024-09-16T13:17:38.648 INFO:teuthology.orchestra.run.smithi098.stdout:2890 write 0x5662404 thru 0x5671ad9 (0xf6d6 bytes) 2024-09-16T13:17:38.654 INFO:teuthology.orchestra.run.smithi098.stdout:2891 read 0x3165cd0 thru 0x316cc21 (0x6f52 bytes) 2024-09-16T13:17:38.655 INFO:teuthology.orchestra.run.smithi098.stdout:2894 write 0x72a9078 thru 0x72b1d78 (0x8d01 bytes) 2024-09-16T13:17:38.661 INFO:teuthology.orchestra.run.smithi098.stdout:2897 read 0x583433f thru 0x583f8c8 (0xb58a bytes) 2024-09-16T13:17:38.661 INFO:teuthology.orchestra.run.smithi098.stdout:2898 write 0x99c4a41 thru 0x99c8f47 (0x4507 bytes) 2024-09-16T13:17:38.667 INFO:teuthology.orchestra.run.smithi098.stdout:2899 read 0x16d6a88 thru 0x16e190d (0xae86 bytes) 2024-09-16T13:17:38.668 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.668 INFO:teuthology.orchestra.run.smithi098.stdout:2903 punch from 0x1d02b01 to 0x1d07a99, (0x4f98 bytes) 2024-09-16T13:17:38.668 INFO:teuthology.orchestra.run.smithi098.stdout:2904 read 0x6413b26 thru 0x641a2ce (0x67a9 bytes) 2024-09-16T13:17:38.668 INFO:teuthology.orchestra.run.smithi098.stdout:2907 write 0x69c229e thru 0x69cbc4f (0x99b2 bytes) 2024-09-16T13:17:38.674 INFO:teuthology.orchestra.run.smithi098.stdout:2908 punch from 0x15ebe0c to 0x15ee01a, (0x220e bytes) 2024-09-16T13:17:38.674 INFO:teuthology.orchestra.run.smithi098.stdout:2909 writesame 0x2cdbb3d thru 0x2cdf6ff data_size 0x1(0x3bc3 bytes) 2024-09-16T13:17:38.690 INFO:teuthology.orchestra.run.smithi098.stdout:2910 read 0xc0b9c13 thru 0xc0c55f2 (0xb9e0 bytes) 2024-09-16T13:17:38.698 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:38.688+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:38.700 INFO:teuthology.orchestra.run.smithi098.stdout:2911 trunc from 0xdb8c547 to 0x88c9850 2024-09-16T13:17:38.711 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:38.711 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:38.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:38.721 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.740 INFO:teuthology.orchestra.run.smithi098.stdout:2913 writesame 0xedc4f38 thru 0xedcdbaa data_size 0x55(0x8c73 bytes) 2024-09-16T13:17:38.741 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.741 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.741 INFO:teuthology.orchestra.run.smithi098.stdout:2918 read 0xade1dbc thru 0xade5372 (0x35b7 bytes) 2024-09-16T13:17:38.741 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.741 INFO:teuthology.orchestra.run.smithi098.stdout:2920 writesame 0xe011de4 thru 0xe01b0c3 data_size 0xa0(0x92e0 bytes) 2024-09-16T13:17:38.756 INFO:teuthology.orchestra.run.smithi098.stdout:2921 write 0xede7613 thru 0xede8fe8 (0x19d6 bytes) 2024-09-16T13:17:38.762 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.762 INFO:teuthology.orchestra.run.smithi098.stdout:2924 write 0x5c24324 thru 0x5c29226 (0x4f03 bytes) 2024-09-16T13:17:38.767 INFO:teuthology.orchestra.run.smithi098.stdout:2925 write 0x1cf5ccd thru 0x1cfc1ea (0x651e bytes) 2024-09-16T13:17:38.774 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.774 INFO:teuthology.orchestra.run.smithi098.stdout:2927 read 0x21fc965 thru 0x21fd893 (0xf2f bytes) 2024-09-16T13:17:38.775 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.775 INFO:teuthology.orchestra.run.smithi098.stdout:2930 write 0x25f0c0b thru 0x25f249a (0x1890 bytes) 2024-09-16T13:17:38.783 INFO:teuthology.orchestra.run.smithi098.stdout:2931 read 0x39e845 thru 0x3a4bf9 (0x63b5 bytes) 2024-09-16T13:17:38.783 INFO:teuthology.orchestra.run.smithi098.stdout:2932 write 0x9bc20e9 thru 0x9bd1368 (0xf280 bytes) 2024-09-16T13:17:38.791 INFO:teuthology.orchestra.run.smithi098.stdout:2934 write 0x45bc795 thru 0x45c600a (0x9876 bytes) 2024-09-16T13:17:38.799 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:38.791+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:38.800 INFO:teuthology.orchestra.run.smithi098.stdout:2935 write 0x2f1b0fa thru 0x2f279e2 (0xc8e9 bytes) 2024-09-16T13:17:38.803 INFO:teuthology.orchestra.run.smithi098.stdout:2937 punch from 0x2261def to 0x2267750, (0x5961 bytes) 2024-09-16T13:17:38.803 INFO:teuthology.orchestra.run.smithi098.stdout:2938 write 0xd32c720 thru 0xd3396fe (0xcfdf bytes) 2024-09-16T13:17:38.809 INFO:teuthology.orchestra.run.smithi098.stdout:2939 write 0x552bee8 thru 0x55355f4 (0x970d bytes) 2024-09-16T13:17:38.822 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:38.822 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:38.826 INFO:teuthology.orchestra.run.smithi098.stdout:2940 trunc from 0xede8fe9 to 0x8ffb7eb 2024-09-16T13:17:38.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:38.854 INFO:teuthology.orchestra.run.smithi098.stdout:2941 trunc from 0x8ffb7eb to 0x3bc4689 2024-09-16T13:17:38.899 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:38.892+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:38.905 INFO:teuthology.orchestra.run.smithi098.stdout:2943 write 0xc43ab0e thru 0xc440277 (0x576a bytes) 2024-09-16T13:17:38.929 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:38.929 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:38.930 INFO:teuthology.orchestra.run.smithi098.stdout:2944 write 0xe3fa160 thru 0xe4015b6 (0x7457 bytes) 2024-09-16T13:17:38.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:38.939 INFO:teuthology.orchestra.run.smithi098.stdout:2945 write 0xd7ad982 thru 0xd7b074b (0x2dca bytes) 2024-09-16T13:17:38.945 INFO:teuthology.orchestra.run.smithi098.stdout:2946 write 0x6c2bb90 thru 0x6c32b6b (0x6fdc bytes) 2024-09-16T13:17:38.951 INFO:teuthology.orchestra.run.smithi098.stdout:2947 write 0x1483685 thru 0x148f2eb (0xbc67 bytes) 2024-09-16T13:17:38.958 INFO:teuthology.orchestra.run.smithi098.stdout:2949 punch from 0x1165347 to 0x1169d81, (0x4a3a bytes) 2024-09-16T13:17:38.958 INFO:teuthology.orchestra.run.smithi098.stdout:2950 writesame 0x33d9035 thru 0x33ddb7a data_size 0x5(0x4b46 bytes) 2024-09-16T13:17:38.966 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:38.966 INFO:teuthology.orchestra.run.smithi098.stdout:2953 punch from 0x20e9b31 to 0x20eba4e, (0x1f1d bytes) 2024-09-16T13:17:38.966 INFO:teuthology.orchestra.run.smithi098.stdout:2954 writesame 0x80ed44d thru 0x80f3496 data_size 0x3a(0x604a bytes) 2024-09-16T13:17:38.972 INFO:teuthology.orchestra.run.smithi098.stdout:2957 punch from 0x9dee73a to 0x9df5fbc, (0x7882 bytes) 2024-09-16T13:17:38.973 INFO:teuthology.orchestra.run.smithi098.stdout:2959 read 0x9f91a92 thru 0x9f93997 (0x1f06 bytes) 2024-09-16T13:17:38.973 INFO:teuthology.orchestra.run.smithi098.stdout:2960 write 0x217e0de thru 0x2189af1 (0xba14 bytes) 2024-09-16T13:17:38.981 INFO:teuthology.orchestra.run.smithi098.stdout:2964 read 0x29fc88b thru 0x2a0ac73 (0xe3e9 bytes) 2024-09-16T13:17:38.982 INFO:teuthology.orchestra.run.smithi098.stdout:2966 punch from 0x5b5eaf8 to 0x5b679dc, (0x8ee4 bytes) 2024-09-16T13:17:38.982 INFO:teuthology.orchestra.run.smithi098.stdout:2967 punch from 0x78f239c to 0x78fd959, (0xb5bd bytes) 2024-09-16T13:17:39.001 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:38.996+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:39.002 INFO:teuthology.orchestra.run.smithi098.stdout:2968 trunc from 0xe4015b7 to 0x5328063 2024-09-16T13:17:39.033 INFO:teuthology.orchestra.run.smithi098.stdout:2969 trunc from 0x5328063 to 0xceb47b 2024-09-16T13:17:39.044 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:39.044 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:39.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:39.071 INFO:teuthology.orchestra.run.smithi098.stdout:2972 read 0x7ad0f9 thru 0x7b8d6c (0xbc74 bytes) 2024-09-16T13:17:39.082 INFO:teuthology.orchestra.run.smithi098.stdout:2973 trunc from 0xceb47b to 0x5454b8b 2024-09-16T13:17:39.092 INFO:teuthology.orchestra.run.smithi098.stdout:2974 write 0x47a8dc1 thru 0x47b82f3 (0xf533 bytes) 2024-09-16T13:17:39.098 INFO:teuthology.orchestra.run.smithi098.stdout:2975 punch from 0x3f9164f to 0x3f99383, (0x7d34 bytes) 2024-09-16T13:17:39.098 INFO:teuthology.orchestra.run.smithi098.stdout:2976 read 0x343b623 thru 0x3447451 (0xbe2f bytes) 2024-09-16T13:17:39.101 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:39.096+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:39.102 INFO:teuthology.orchestra.run.smithi098.stdout:2977 trunc from 0x5454b8b to 0x6de0a81 2024-09-16T13:17:39.117 INFO:teuthology.orchestra.run.smithi098.stdout:2979 trunc from 0x6de0a81 to 0x475e03b 2024-09-16T13:17:39.147 INFO:teuthology.orchestra.run.smithi098.stdout:2980 write 0x5c07bdc thru 0x5c11720 (0x9b45 bytes) 2024-09-16T13:17:39.155 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:39.155 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:39.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:39.183 INFO:teuthology.orchestra.run.smithi098.stdout:2981 write 0xe1964f2 thru 0xe1a618c (0xfc9b bytes) 2024-09-16T13:17:39.190 INFO:teuthology.orchestra.run.smithi098.stdout:2982 writesame 0xa946fba thru 0xa956cec data_size 0x1(0xfd33 bytes) 2024-09-16T13:17:39.202 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:39.192+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:39.249 INFO:teuthology.orchestra.run.smithi098.stdout:2983 punch from 0xaa12e09 to 0xaa13f1c, (0x1113 bytes) 2024-09-16T13:17:39.249 INFO:teuthology.orchestra.run.smithi098.stdout:2984 read 0xcd0056b thru 0xcd005de (0x74 bytes) 2024-09-16T13:17:39.254 INFO:teuthology.orchestra.run.smithi098.stdout:2985 trunc from 0xe1a618d to 0xbbdc9fe 2024-09-16T13:17:39.261 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:39.262 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:39.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:39.283 INFO:teuthology.orchestra.run.smithi098.stdout:2987 writesame 0x19f1eaa thru 0x19f4bbc data_size 0x1(0x2d13 bytes) 2024-09-16T13:17:39.303 INFO:teuthology.orchestra.run.smithi098.stdout:2988 read 0x16e07ed thru 0x16f015b (0xf96f bytes) 2024-09-16T13:17:39.303 INFO:teuthology.orchestra.run.smithi098.stdout:2989 read 0x6a6f3a9 thru 0x6a7e9bd (0xf615 bytes) 2024-09-16T13:17:39.304 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:39.296+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:39.304 INFO:teuthology.orchestra.run.smithi098.stdout:2990 write 0x9dc79b2 thru 0x9dcb49f (0x3aee bytes) 2024-09-16T13:17:39.311 INFO:teuthology.orchestra.run.smithi098.stdout:2991 read 0x5b51845 thru 0x5b5ed7f (0xd53b bytes) 2024-09-16T13:17:39.311 INFO:teuthology.orchestra.run.smithi098.stdout:2992 punch from 0x1a5a341 to 0x1a62bf1, (0x88b0 bytes) 2024-09-16T13:17:39.311 INFO:teuthology.orchestra.run.smithi098.stdout:2993 read 0x9e59b57 thru 0x9e66dbd (0xd267 bytes) 2024-09-16T13:17:39.311 INFO:teuthology.orchestra.run.smithi098.stdout:2995 read 0x279f37a thru 0x27a21c2 (0x2e49 bytes) 2024-09-16T13:17:39.329 INFO:teuthology.orchestra.run.smithi098.stdout:2996 trunc from 0xbbdc9fe to 0x35ca575 2024-09-16T13:17:39.348 INFO:teuthology.orchestra.run.smithi098.stdout:2998 write 0x1eff9bf thru 0x1f051d4 (0x5816 bytes) 2024-09-16T13:17:39.353 INFO:teuthology.orchestra.run.smithi098.stdout:2999 write 0x319bb5c thru 0x31a48ca (0x8d6f bytes) 2024-09-16T13:17:39.359 INFO:teuthology.orchestra.run.smithi098.stdout:3000 punch from 0x2b045e1 to 0x2b09f4f, (0x596e bytes) 2024-09-16T13:17:39.370 INFO:teuthology.orchestra.run.smithi098.stdout:3001 trunc from 0x35ca575 to 0x8e4c445 2024-09-16T13:17:39.380 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:39.380 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:39.391 INFO:teuthology.orchestra.run.smithi098.stdout:3002 trunc from 0x8e4c445 to 0xe1550e3 2024-09-16T13:17:39.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:39.417 INFO:teuthology.orchestra.run.smithi098.stdout:3007 write 0xe544c21 thru 0xe549d63 (0x5143 bytes) 2024-09-16T13:17:39.417 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:39.408+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:39.424 INFO:teuthology.orchestra.run.smithi098.stdout:3008 punch from 0xcae7e75 to 0xcaf2dce, (0xaf59 bytes) 2024-09-16T13:17:39.424 INFO:teuthology.orchestra.run.smithi098.stdout:3010 punch from 0x85797c8 to 0x8587ad3, (0xe30b bytes) 2024-09-16T13:17:39.425 INFO:teuthology.orchestra.run.smithi098.stdout:3011 write 0x2f4d0ba thru 0x2f5b2bb (0xe202 bytes) 2024-09-16T13:17:39.430 INFO:teuthology.orchestra.run.smithi098.stdout:3012 writesame 0x55b65e6 thru 0x55b7225 data_size 0x1(0xc40 bytes) 2024-09-16T13:17:39.453 INFO:teuthology.orchestra.run.smithi098.stdout:3014 trunc from 0xe549d64 to 0x894d917 2024-09-16T13:17:39.491 INFO:teuthology.orchestra.run.smithi098.stdout:3015 write 0xafc27e6 thru 0xafca12e (0x7949 bytes) 2024-09-16T13:17:39.498 INFO:teuthology.orchestra.run.smithi098.stdout:3016 writesame 0xa1476a7 thru 0xa155774 data_size 0x19(0xe0ce bytes) 2024-09-16T13:17:39.505 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:39.499+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:39.508 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:39.508 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:39.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:39.521 INFO:teuthology.orchestra.run.smithi098.stdout:3017 trunc from 0xafca12f to 0x4b58a8f 2024-09-16T13:17:39.546 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:17:39.574 INFO:teuthology.orchestra.run.smithi098.stdout:3019 write 0xd0e847b thru 0xd0eec19 (0x679f bytes) 2024-09-16T13:17:39.581 INFO:teuthology.orchestra.run.smithi098.stdout:3020 write 0x32400a3 thru 0x3247d6e (0x7ccc bytes) 2024-09-16T13:17:39.587 INFO:teuthology.orchestra.run.smithi098.stdout:3021 read 0xb2340a0 thru 0xb239042 (0x4fa3 bytes) 2024-09-16T13:17:39.587 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:39.587 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:39.587 INFO:teuthology.orchestra.run.smithi098.stdout:3024 write 0x8218996 thru 0x821b001 (0x266c bytes) 2024-09-16T13:17:39.592 INFO:teuthology.orchestra.run.smithi098.stdout:3025 read 0x373cdd thru 0x37d870 (0x9b94 bytes) 2024-09-16T13:17:39.606 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:39.599+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:39.606 INFO:teuthology.orchestra.run.smithi098.stdout:3027 write 0xd2ea25f thru 0xd2ed339 (0x30db bytes) 2024-09-16T13:17:39.616 INFO:teuthology.orchestra.run.smithi098.stdout:3028 trunc from 0xd2ed33a to 0xcd23b45 2024-09-16T13:17:39.638 INFO:teuthology.orchestra.run.smithi098.stdout:3030 trunc from 0xcd23b45 to 0x9781410 2024-09-16T13:17:39.646 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:39.646 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:39.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:39.660 INFO:teuthology.orchestra.run.smithi098.stdout:3031 write 0x16dbed6 thru 0x16e31aa (0x72d5 bytes) 2024-09-16T13:17:39.667 INFO:teuthology.orchestra.run.smithi098.stdout:3033 read 0x307c8ad thru 0x307cd3a (0x48e bytes) 2024-09-16T13:17:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:3035 trunc from 0x9781410 to 0x883e7b3 2024-09-16T13:17:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:3036 write 0xbec482 thru 0xbec743 (0x2c2 bytes) 2024-09-16T13:17:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:3037 punch from 0x6222543 to 0x6231536, (0xeff3 bytes) 2024-09-16T13:17:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:3038 writesame 0x29d41d6 thru 0x29dc712 data_size 0x1(0x853d bytes) 2024-09-16T13:17:39.707 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:39.699+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:39.720 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:39.720 INFO:teuthology.orchestra.run.smithi098.stdout:3040 read 0x1659206 thru 0x165e1dd (0x4fd8 bytes) 2024-09-16T13:17:39.720 INFO:teuthology.orchestra.run.smithi098.stdout:3041 write 0x252d9b1 thru 0x2531044 (0x3694 bytes) 2024-09-16T13:17:39.728 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:39.729 INFO:teuthology.orchestra.run.smithi098.stdout:3043 read 0x3fbf1f7 thru 0x3fc4006 (0x4e10 bytes) 2024-09-16T13:17:39.729 INFO:teuthology.orchestra.run.smithi098.stdout:3044 writesame 0x5e71d58 thru 0x5e7d54f data_size 0x74(0xb7f8 bytes) 2024-09-16T13:17:39.735 INFO:teuthology.orchestra.run.smithi098.stdout:3045 write 0x60de13b thru 0x60de4f7 (0x3bd bytes) 2024-09-16T13:17:39.737 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:39.737 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:39.738 INFO:teuthology.orchestra.run.smithi098.stdout:3049 trunc from 0x883e7b3 to 0x822542b 2024-09-16T13:17:39.754 INFO:teuthology.orchestra.run.smithi098.stdout:3050 read 0x2d5fdde thru 0x2d629de (0x2c01 bytes) 2024-09-16T13:17:39.758 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:39.758 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:39.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:3051 write 0xec04e08 thru 0xec0c7dc (0x79d5 bytes) 2024-09-16T13:17:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:3053 read 0x2dad10a thru 0x2db7681 (0xa578 bytes) 2024-09-16T13:17:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:3054 punch from 0x3625283 to 0x362b2e2, (0x605f bytes) 2024-09-16T13:17:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:3055 read 0x3cbc4ba thru 0x3cbd0d6 (0xc1d bytes) 2024-09-16T13:17:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:3056 write 0xbc1878f thru 0xbc1cfd2 (0x4844 bytes) 2024-09-16T13:17:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:3057 trunc from 0xec0c7dd to 0xc821aa4 2024-09-16T13:17:39.808 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:39.800+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:39.818 INFO:teuthology.orchestra.run.smithi098.stdout:3061 write 0x4dcf53e thru 0x4ddcb6b (0xd62e bytes) 2024-09-16T13:17:39.827 INFO:teuthology.orchestra.run.smithi098.stdout:3062 trunc from 0xc821aa4 to 0xd58393b 2024-09-16T13:17:39.841 INFO:teuthology.orchestra.run.smithi098.stdout:3064 write 0xae0d1ec thru 0xae10c74 (0x3a89 bytes) 2024-09-16T13:17:39.845 INFO:teuthology.orchestra.run.smithi098.stdout:3067 write 0xb8cb80e thru 0xb8d8177 (0xc96a bytes) 2024-09-16T13:17:39.853 INFO:teuthology.orchestra.run.smithi098.stdout:3070 write 0xc0de74f thru 0xc0e22cc (0x3b7e bytes) 2024-09-16T13:17:39.860 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:39.861 INFO:teuthology.orchestra.run.smithi098.stdout:3073 read 0x9b9c3dc thru 0x9ba6165 (0x9d8a bytes) 2024-09-16T13:17:39.861 INFO:teuthology.orchestra.run.smithi098.stdout:3074 writesame 0xb53577c thru 0xb53d161 data_size 0xe(0x79e6 bytes) 2024-09-16T13:17:39.870 INFO:teuthology.orchestra.run.smithi098.stdout:3076 punch from 0x9076cc3 to 0x907a0d3, (0x3410 bytes) 2024-09-16T13:17:39.887 INFO:teuthology.orchestra.run.smithi098.stdout:3077 writesame 0xed2bc8b thru 0xed3758f data_size 0x1(0xb905 bytes) 2024-09-16T13:17:39.887 INFO:teuthology.orchestra.run.smithi098.stdout:3078 writesame 0xaf6dba6 thru 0xaf70c0f data_size 0x1(0x306a bytes) 2024-09-16T13:17:39.892 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:39.892 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:39.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:39.904 INFO:teuthology.orchestra.run.smithi098.stdout:3079 read 0xb02e065 thru 0xb02fd03 (0x1c9f bytes) 2024-09-16T13:17:39.904 INFO:teuthology.orchestra.run.smithi098.stdout:3080 read 0x59a7cfc thru 0x59af499 (0x779e bytes) 2024-09-16T13:17:39.904 INFO:teuthology.orchestra.run.smithi098.stdout:3082 punch from 0x505d0a4 to 0x505f054, (0x1fb0 bytes) 2024-09-16T13:17:39.904 INFO:teuthology.orchestra.run.smithi098.stdout:3083 write 0x7ab3297 thru 0x7ab33d9 (0x143 bytes) 2024-09-16T13:17:39.907 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:39.907 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:39.907 INFO:teuthology.orchestra.run.smithi098.stdout:3086 writesame 0x4d2bb25 thru 0x4d32296 data_size 0x1(0x6772 bytes) 2024-09-16T13:17:39.909 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:39.903+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:39.935 INFO:teuthology.orchestra.run.smithi098.stdout:3088 read 0x6642e1b thru 0x6645eab (0x3091 bytes) 2024-09-16T13:17:39.935 INFO:teuthology.orchestra.run.smithi098.stdout:3090 write 0x5ad06ed thru 0x5ad2642 (0x1f56 bytes) 2024-09-16T13:17:39.955 INFO:teuthology.orchestra.run.smithi098.stdout:3092 write 0xed7be7a thru 0xed858a2 (0x9a29 bytes) 2024-09-16T13:17:39.965 INFO:teuthology.orchestra.run.smithi098.stdout:3093 trunc from 0xed858a3 to 0xe7845cc 2024-09-16T13:17:39.983 INFO:teuthology.orchestra.run.smithi098.stdout:3094 read 0x8dcc259 thru 0x8dccf68 (0xd10 bytes) 2024-09-16T13:17:39.984 INFO:teuthology.orchestra.run.smithi098.stdout:3096 punch from 0xd38dcdc to 0xd392ba0, (0x4ec4 bytes) 2024-09-16T13:17:39.984 INFO:teuthology.orchestra.run.smithi098.stdout:3097 read 0x7da3ba8 thru 0x7da9bf4 (0x604d bytes) 2024-09-16T13:17:39.984 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:39.984 INFO:teuthology.orchestra.run.smithi098.stdout:3099 read 0x13e2aa2 thru 0x13e875a (0x5cb9 bytes) 2024-09-16T13:17:39.984 INFO:teuthology.orchestra.run.smithi098.stdout:3100 read 0x67f9a00 thru 0x67fd8f8 (0x3ef9 bytes) 2024-09-16T13:17:39.985 INFO:teuthology.orchestra.run.smithi098.stdout:3101 writesame 0xac7665b thru 0xac77e4f data_size 0x1(0x17f5 bytes) 2024-09-16T13:17:39.998 INFO:teuthology.orchestra.run.smithi098.stdout:3103 write 0x129bc05 thru 0x129cec7 (0x12c3 bytes) 2024-09-16T13:17:40.001 INFO:teuthology.orchestra.run.smithi098.stdout:3104 read 0x931d9a0 thru 0x9321441 (0x3aa2 bytes) 2024-09-16T13:17:40.001 INFO:teuthology.orchestra.run.smithi098.stdout:3105 read 0xc53bf5d thru 0xc53e882 (0x2926 bytes) 2024-09-16T13:17:40.005 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:40.005 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:40.010 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:40.003+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:40.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:40.022 INFO:teuthology.orchestra.run.smithi098.stdout:3107 trunc from 0xe7845cc to 0x2283166 2024-09-16T13:17:40.048 INFO:teuthology.orchestra.run.smithi098.stdout:3109 read 0x2d81e2 thru 0x2dd315 (0x5134 bytes) 2024-09-16T13:17:40.070 INFO:teuthology.orchestra.run.smithi098.stdout:3110 write 0x8d7b4d8 thru 0x8d85dc7 (0xa8f0 bytes) 2024-09-16T13:17:40.079 INFO:teuthology.orchestra.run.smithi098.stdout:3111 punch from 0x15ecec1 to 0x15f9e3c, (0xcf7b bytes) 2024-09-16T13:17:40.086 INFO:teuthology.orchestra.run.smithi098.stdout:3112 trunc from 0x8d85dc8 to 0x45cef15 2024-09-16T13:17:40.108 INFO:teuthology.orchestra.run.smithi098.stdout:3113 trunc from 0x45cef15 to 0x5253521 2024-09-16T13:17:40.111 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:40.100+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:40.118 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:40.118 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:40.122 INFO:teuthology.orchestra.run.smithi098.stdout:3114 writesame 0x3ed33d7 thru 0x3ed4f8f data_size 0x1(0x1bb9 bytes) 2024-09-16T13:17:40.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:40.140 INFO:teuthology.orchestra.run.smithi098.stdout:3116 trunc from 0x5253521 to 0x7d1ed78 2024-09-16T13:17:40.154 INFO:teuthology.orchestra.run.smithi098.stdout:3117 clone 17 order 24 su 8388608 sc 13 2024-09-16T13:17:40.212 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:40.203+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:40.237 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:40.237 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:40.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:40.312 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:40.304+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:40.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:40.406 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:40.406 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:40.414 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:40.411+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:40.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:40.456 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:40.475 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:40.495 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:40.515 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.515 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1264 3.4 e27/23 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head v 27'4, mlcod=26'2)", 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.008553001000003, 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026410209999999999, 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501250+0000", 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2410000000000001e-06 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501259+0000", 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:40.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501277+0000", 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8683999999999999e-05 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501288+0000", 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.065e-05 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501562+0000", 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027421200000000002 2024-09-16T13:17:40.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501618+0000", 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6045000000000003e-05 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503841+0000", 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022230829999999998 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503890+0000", 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8526000000000003e-05 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.502409313000001, 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:17:40.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.524 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:17:40.525 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:40.525 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.501623813999998, 2024-09-16T13:17:40.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:17:40.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.525 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:40.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.500456892000003, 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:17:40.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:17:40.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:17:40.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.497582260999998, 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:17:40.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.494568520999998, 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:17:40.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:17:40.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:17:40.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:17:40.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:17:40.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:17:40.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:17:40.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:17:40.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:17:40.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:17:40.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.469364317999997, 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.544 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 35.703215428999997, 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:17:40.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:17:40.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5115 3.4 e34/23 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head v 34'14, mlcod=31'12)", 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.718167323000003, 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029400680000000001, 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791635+0000", 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4200000000000005e-07 2024-09-16T13:17:40.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791643+0000", 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.157e-06 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791660+0000", 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6246e-05 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791666+0000", 2024-09-16T13:17:40.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791743+0000", 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7000000000000001e-05 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791756+0000", 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2729e-05 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794523+0000", 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027675949999999999 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794575+0000", 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1082000000000003e-05 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:40.557 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.451113125999999, 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.561 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.450503686000001, 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:17:40.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:17:40.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6963 3.4 e37/23 3:2713f365:::rbd_data.112d6fb8d387.0000000000000133:head v 37'31, mlcod=37'29)", 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.953939536, 2024-09-16T13:17:40.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027598190000000002, 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555864+0000", 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555882+0000", 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7816e-05 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555909+0000", 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7838999999999999e-05 2024-09-16T13:17:40.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555923+0000", 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3577000000000001e-05 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556083+0000", 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015987299999999999 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556132+0000", 2024-09-16T13:17:40.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8655000000000001e-05 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558539+0000", 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024076969999999999 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558622+0000", 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2777999999999999e-05 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.953219871999998, 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:17:40.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.952119056000001, 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.575 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.951979416, 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:17:40.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.951478886, 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:17:40.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:17:40.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:10843 3.0 e40/23 3:039a6044:::rbd_data.112ddef65b5a.0000000000000050:head v 40'68, mlcod=39'65)", 2024-09-16T13:17:40.588 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.242323+0000", 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.267478468, 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025010879999999998, 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242323+0000", 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242323+0000", 2024-09-16T13:17:40.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242324+0000", 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242332+0000", 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5529999999999993e-06 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242354+0000", 2024-09-16T13:17:40.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1336999999999999e-05 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242362+0000", 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9859999999999994e-06 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242409+0000", 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7416000000000001e-05 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.242431+0000", 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2291000000000001e-05 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.244771+0000", 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023402280000000002 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.244824+0000", 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 22.9563095, 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:40.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:17:40.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:40.597 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.699895658999999, 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.599 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.601 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 12.087332551999999, 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:40.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:17:40.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:17:40.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.607 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.607 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.607 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:40.608 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:40.507+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:40.608 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:40.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:40.610 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:40.610 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:40.610 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:40.610 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:40.610 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:40.611 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:40.615 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:40.607+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:40.666 INFO:teuthology.orchestra.run.smithi098.stdout:flattening image image_client.0-clone16 2024-09-16T13:17:40.709 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:40.709 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:40.710 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:40.717 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:40.711+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:40.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:40.818 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:40.811+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:40.821 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:40.821 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:40.821 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #15, image image_client.0-clone15 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent16 2024-09-16T13:17:40.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:40.920 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:40.911+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:40.939 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:40.939 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:40.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:40.939 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:40.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.939 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5101 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:40.939 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:40.939 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.154585617000002, 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595248, 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5101 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:40.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:40.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777333+0000", 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1949999999999996e-06 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777342+0000", 2024-09-16T13:17:40.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6950000000000006e-06 2024-09-16T13:17:40.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777348+0000", 2024-09-16T13:17:40.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:17:40.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785241+0000", 2024-09-16T13:17:40.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078929329999999995 2024-09-16T13:17:40.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785391+0000", 2024-09-16T13:17:40.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150113 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785969+0000", 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057822899999999998 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789745+0000", 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037756529999999999 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793823+0000", 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040786299999999998 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793880+0000", 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7157999999999997e-05 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793924+0000", 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3407e-05 2024-09-16T13:17:40.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5102 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.154538324999997, 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016663710000000002, 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:40.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5102 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777377+0000", 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4600000000000003e-07 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777381+0000", 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5290000000000002e-06 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777393+0000", 2024-09-16T13:17:40.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2001000000000001e-05 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777401+0000", 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3869999999999997e-06 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786753+0000", 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093523649999999996 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786860+0000", 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106921 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787286+0000", 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042651800000000003 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793815+0000", 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065290579999999999 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793932+0000", 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116688 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794002+0000", 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0313000000000003e-05 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794040+0000", 2024-09-16T13:17:40.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7083999999999999e-05 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5103 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.154477438999997, 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017505544000000001, 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:40.954 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5103 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777438+0000", 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.78e-07 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777442+0000", 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5950000000000001e-06 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777453+0000", 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0372e-05 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777458+0000", 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6559999999999997e-06 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788034+0000", 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010575753 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788111+0000", 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7038000000000003e-05 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788463+0000", 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000352028 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794263+0000", 2024-09-16T13:17:40.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005800342 2024-09-16T13:17:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794837+0000", 2024-09-16T13:17:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057334300000000003 2024-09-16T13:17:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794897+0000", 2024-09-16T13:17:40.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0690000000000003e-05 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794942+0000", 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4848999999999997e-05 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.960 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5104 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.154433031000003, 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017694142999999999, 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5104 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777482+0000", 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:17:40.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777487+0000", 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6639999999999997e-06 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777496+0000", 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2890000000000003e-06 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777501+0000", 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2730000000000002e-06 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788951+0000", 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011449429000000001 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789007+0000", 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5643999999999998e-05 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789205+0000", 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019820800000000001 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794268+0000", 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050636850000000001 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795094+0000", 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082569700000000002 2024-09-16T13:17:40.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795139+0000", 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5055000000000001e-05 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795175+0000", 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6149000000000002e-05 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5105 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.15437069, 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017729482000000001, 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5105 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777544+0000", 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1100000000000004e-07 2024-09-16T13:17:40.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777549+0000", 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777561+0000", 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1524e-05 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777566+0000", 2024-09-16T13:17:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.558e-06 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789781+0000", 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012214981 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789870+0000", 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9171000000000004e-05 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790313+0000", 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044255299999999999 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794273+0000", 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039596129999999998 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795213+0000", 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094086099999999998 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795239+0000", 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5324e-05 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795273+0000", 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4198e-05 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.972 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5106 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.154325546000003, 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018621867, 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5106 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777590+0000", 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777594+0000", 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6750000000000001e-06 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777604+0000", 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0076e-05 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777610+0000", 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7529999999999998e-06 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790893+0000", 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013282845 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790999+0000", 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106289 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:40.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791493+0000", 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049357800000000005 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794662+0000", 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031691409999999999 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796146+0000", 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00148375 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796176+0000", 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0349e-05 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796211+0000", 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4530000000000003e-05 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5107 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.154294284999999, 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018745067000000001, 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5107 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777621+0000", 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-07 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777624+0000", 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7950000000000001e-06 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777633+0000", 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5979999999999997e-06 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777638+0000", 2024-09-16T13:17:40.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.989e-06 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792024+0000", 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014385692 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792104+0000", 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0567000000000003e-05 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792340+0000", 2024-09-16T13:17:40.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023624200000000001 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794667+0000", 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023269990000000002 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796304+0000", 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001636981 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796331+0000", 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6492999999999999e-05 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796365+0000", 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4102e-05 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5108 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:40.984 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.154240455999997, 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019090617000000001, 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5108 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:40.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777677+0000", 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7879999999999998e-06 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777686+0000", 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7679999999999997e-06 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777689+0000", 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.123e-06 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792751+0000", 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01506158 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792855+0000", 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010456 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793305+0000", 2024-09-16T13:17:40.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045006100000000003 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794671+0000", 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001366304 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796393+0000", 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001721272 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796733+0000", 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000340611 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796764+0000", 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.099e-05 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.990 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5208 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.052481004999997, 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014882177999999999, 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5208 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879434+0000", 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.322e-06 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879443+0000", 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879459+0000", 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6133999999999998e-05 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879467+0000", 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2400000000000007e-06 2024-09-16T13:17:40.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888339+0000", 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088721619999999994 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888461+0000", 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121341 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:40.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888819+0000", 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035782799999999998 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891271+0000", 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002452872 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894224+0000", 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029522910000000001 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894275+0000", 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1647999999999997e-05 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894315+0000", 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9993999999999997e-05 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.996 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5210 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.051925816999997, 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01466423, 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5210 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:40.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879989+0000", 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8800000000000001e-07 2024-09-16T13:17:40.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879995+0000", 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880017+0000", 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2101000000000001e-05 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880024+0000", 2024-09-16T13:17:40.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9240000000000001e-06 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890250+0000", 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010226012 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890344+0000", 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.446e-05 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:41.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890770+0000", 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042575099999999998 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894344+0000", 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003574163 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894549+0000", 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020477599999999999 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894606+0000", 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6891000000000001e-05 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894653+0000", 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6867999999999999e-05 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.002 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5211 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.051881754, 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014800604, 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5211 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:41.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880033+0000", 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.06e-07 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880039+0000", 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880052+0000", 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-05 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880059+0000", 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3090000000000001e-06 2024-09-16T13:17:41.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891316+0000", 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011257118999999999 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891374+0000", 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8020999999999999e-05 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:41.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891577+0000", 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020371699999999999 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894348+0000", 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027708300000000002 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894726+0000", 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037819399999999999 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894784+0000", 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7667999999999997e-05 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894833+0000", 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8936e-05 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5212 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.051526379999999, 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014645704000000001, 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5212 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880389+0000", 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1680000000000001e-06 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880395+0000", 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880408+0000", 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.308e-05 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880414+0000", 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.105e-06 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892037+0000", 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011622402 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892116+0000", 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9939999999999997e-05 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892597+0000", 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048076500000000002 2024-09-16T13:17:41.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894492+0000", 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001895246 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894921+0000", 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000428114 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894985+0000", 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4777999999999995e-05 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895034+0000", 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8133999999999997e-05 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5213 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:41.014 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.051393230000002, 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014632608, 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5213 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880522+0000", 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2600000000000001e-07 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:41.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880527+0000", 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880538+0000", 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917999999999999e-05 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880543+0000", 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.361e-06 2024-09-16T13:17:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893069+0000", 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012525497999999999 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893152+0000", 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2805000000000004e-05 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893635+0000", 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048315100000000002 2024-09-16T13:17:41.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894852+0000", 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001216806 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895075+0000", 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022300299999999999 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895122+0000", 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7077999999999999e-05 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895154+0000", 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1919999999999999e-05 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5214 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:41.020 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.051238579, 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017785977000000001, 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5214 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:41.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:41.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880681+0000", 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880689+0000", 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3610000000000001e-06 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880694+0000", 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5809999999999998e-06 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895186+0000", 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014492430000000001 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895311+0000", 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012518300000000001 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895760+0000", 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044864900000000002 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898337+0000", 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025765229999999998 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898354+0000", 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7538999999999999e-05 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898406+0000", 2024-09-16T13:17:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2054e-05 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898462+0000", 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5404e-05 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5215 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:41.026 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.050973745, 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018191319000000001, 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5215 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:41.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880941+0000", 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880946+0000", 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3710000000000002e-06 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880955+0000", 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1230000000000006e-06 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880959+0000", 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:41.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896782+0000", 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015822716000000001 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896876+0000", 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3667999999999998e-05 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:41.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.897289+0000", 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041357399999999999 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898474+0000", 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011847209999999999 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899036+0000", 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056215699999999998 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899090+0000", 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3662999999999998e-05 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899132+0000", 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1814000000000003e-05 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.032 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5282 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.982301630999999, 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014924988, 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5282 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:41.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.53e-07 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949635+0000", 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2422999999999999e-05 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949641+0000", 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.896e-06 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949644+0000", 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5270000000000001e-06 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959656+0000", 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010012378000000001 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959747+0000", 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1273000000000005e-05 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960045+0000", 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029733 2024-09-16T13:17:41.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963196+0000", 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031512620000000002 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964453+0000", 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001257357 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964502+0000", 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9044e-05 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964538+0000", 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5244999999999999e-05 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5283 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.982266477000003, 2024-09-16T13:17:41.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015065762, 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5283 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949650+0000", 2024-09-16T13:17:41.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.736e-06 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949653+0000", 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2969999999999999e-06 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949655+0000", 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1890000000000001e-06 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960643+0000", 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010988015 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960732+0000", 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9097000000000004e-05 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961195+0000", 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000462292 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963208+0000", 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002013641 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964585+0000", 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013765769999999999 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964669+0000", 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3863000000000007e-05 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964713+0000", 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4706999999999997e-05 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5284 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.982256608, 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015152802, 2024-09-16T13:17:41.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:41.045 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:41.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:41.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5284 2024-09-16T13:17:41.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.045 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:41.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5899999999999998e-07 2024-09-16T13:17:41.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949659+0000", 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.601e-06 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949672+0000", 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2364999999999999e-05 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949674+0000", 2024-09-16T13:17:41.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961869+0000", 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012194903999999999 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961971+0000", 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102154 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.962502+0000", 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00053037500000000003 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964751+0000", 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022487800000000001 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964755+0000", 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:41.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964789+0000", 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4054000000000003e-05 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964810+0000", 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0922000000000001e-05 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5285 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.982237689999998, 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015553022, 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5285 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:41.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949677+0000", 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6899999999999999e-07 2024-09-16T13:17:41.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949679+0000", 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0090000000000002e-06 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949682+0000", 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2550000000000002e-06 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949685+0000", 2024-09-16T13:17:41.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.621e-06 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963409+0000", 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013724485999999999 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963497+0000", 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7560999999999997e-05 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963933+0000", 2024-09-16T13:17:41.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043592700000000002 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964820+0000", 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088772299999999998 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965165+0000", 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000344426 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965201+0000", 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5967000000000003e-05 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965230+0000", 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11223 3.8 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 33927~20122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:41.056 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.471065360000001, 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019333487, 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11223 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:41.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460851+0000", 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5730000000000002e-06 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:41.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464824+0000", 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039725730000000001 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464861+0000", 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7233000000000001e-05 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464872+0000", 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209000000000001e-05 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464987+0000", 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011423 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465188+0000", 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020091600000000001 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465929+0000", 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00074102 2024-09-16T13:17:41.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.475697+0000", 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009768367 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480068+0000", 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043713290000000002 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480143+0000", 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5000999999999996e-05 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480182+0000", 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9035999999999997e-05 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:41.062 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:41.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:41.064 INFO:teuthology.orchestra.run.smithi098.stdout:3119 trunc from 0x7d1ed78 to 0x501111b 2024-09-16T13:17:41.065 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:41.056+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:41.100 INFO:teuthology.orchestra.run.smithi098.stdout:3120 write 0x707c44d thru 0x707e476 (0x202a bytes) 2024-09-16T13:17:41.102 INFO:teuthology.orchestra.run.smithi098.stdout:3121 read 0x655f3fe thru 0x6565bc7 (0x67ca bytes) 2024-09-16T13:17:41.103 INFO:teuthology.orchestra.run.smithi098.stdout:3123 writesame 0x4e637e9 thru 0x4e6f896 data_size 0x3(0xc0ae bytes) 2024-09-16T13:17:41.149 INFO:teuthology.orchestra.run.smithi098.stdout:3125 write 0xb01e8c6 thru 0xb025371 (0x6aac bytes) 2024-09-16T13:17:41.152 INFO:teuthology.orchestra.run.smithi098.stdout:3127 punch from 0x5483274 to 0x548fb46, (0xc8d2 bytes) 2024-09-16T13:17:41.152 INFO:teuthology.orchestra.run.smithi098.stdout:3129 read 0x9146a06 thru 0x9153841 (0xce3c bytes) 2024-09-16T13:17:41.159 INFO:teuthology.orchestra.run.smithi098.stdout:3130 trunc from 0xb025372 to 0xe2a8059 2024-09-16T13:17:41.165 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:41.156+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:41.166 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:41.166 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:41.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:41.178 INFO:teuthology.orchestra.run.smithi098.stdout:3131 trunc from 0xe2a8059 to 0x9d536a5 2024-09-16T13:17:41.217 INFO:teuthology.orchestra.run.smithi098.stdout:3134 write 0xbdf2f1e thru 0xbdfb1f6 (0x82d9 bytes) 2024-09-16T13:17:41.221 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:41.223 INFO:teuthology.orchestra.run.smithi098.stdout:3136 trunc from 0xbdfb1f7 to 0xac55030 2024-09-16T13:17:41.246 INFO:teuthology.orchestra.run.smithi098.stdout:3138 trunc from 0xac55030 to 0xcdb0bc2 2024-09-16T13:17:41.259 INFO:teuthology.orchestra.run.smithi098.stdout:3139 read 0xba87490 thru 0xba94c9f (0xd810 bytes) 2024-09-16T13:17:41.260 INFO:teuthology.orchestra.run.smithi098.stdout:3140 write 0x8dc05e6 thru 0x8dcb7a6 (0xb1c1 bytes) 2024-09-16T13:17:41.266 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:41.259+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:41.270 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:41.270 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:41.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:41.289 INFO:teuthology.orchestra.run.smithi098.stdout:3142 writesame 0xed686ad thru 0xed68e24 data_size 0x4(0x778 bytes) 2024-09-16T13:17:41.290 INFO:teuthology.orchestra.run.smithi098.stdout:3143 read 0x43242e7 thru 0x432a6f0 (0x640a bytes) 2024-09-16T13:17:41.291 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:41.319 INFO:teuthology.orchestra.run.smithi098.stdout:3145 trunc from 0xed68e25 to 0x14b95fd 2024-09-16T13:17:41.362 INFO:teuthology.orchestra.run.smithi098.stdout:3146 trunc from 0x14b95fd to 0x7992f9b 2024-09-16T13:17:41.366 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:41.359+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:41.373 INFO:teuthology.orchestra.run.smithi098.stdout:3147 read 0x4e40475 thru 0x4e42092 (0x1c1e bytes) 2024-09-16T13:17:41.373 INFO:teuthology.orchestra.run.smithi098.stdout:3149 write 0x76c59fd thru 0x76cd521 (0x7b25 bytes) 2024-09-16T13:17:41.376 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:41.377 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:41.377 INFO:teuthology.orchestra.run.smithi098.stdout:3152 read 0x3741845 thru 0x374b264 (0x9a20 bytes) 2024-09-16T13:17:41.377 INFO:teuthology.orchestra.run.smithi098.stdout:3153 writesame 0x21d5a42 thru 0x21d5c71 data_size 0xa(0x230 bytes) 2024-09-16T13:17:41.378 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:41.378 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:41.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8369 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.316492832000002, 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020058919000000001, 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.379 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8369 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056335+0000", 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056338+0000", 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:17:41.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056344+0000", 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8150000000000002e-06 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056348+0000", 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.070951+0000", 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014603055 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071060+0000", 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109462 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071508+0000", 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044824200000000001 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074304+0000", 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027951719999999998 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076331+0000", 2024-09-16T13:17:41.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020276529999999999 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076369+0000", 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7557e-05 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076393+0000", 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4445999999999999e-05 2024-09-16T13:17:41.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8370 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.316475514, 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021240851000000002, 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8370 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056355+0000", 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.667e-06 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056360+0000", 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6620000000000002e-06 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056366+0000", 2024-09-16T13:17:41.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072672+0000", 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016306157000000002 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072746+0000", 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3648999999999999e-05 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.073364+0000", 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061885499999999997 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076445+0000", 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003080396 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077482+0000", 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001036709 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077535+0000", 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3211000000000003e-05 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077592+0000", 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7664000000000002e-05 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8371 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.316446718999998, 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021386721000000001, 2024-09-16T13:17:41.393 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8371 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056381+0000", 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056385+0000", 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056391+0000", 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6959999999999999e-06 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056396+0000", 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2549999999999999e-06 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074355+0000", 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017959098999999999 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074503+0000", 2024-09-16T13:17:41.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014797200000000001 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074955+0000", 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045213300000000002 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077607+0000", 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026519500000000001 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077660+0000", 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077725+0000", 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4991000000000002e-05 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077767+0000", 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8372 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.316425698, 2024-09-16T13:17:41.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021557531000000001, 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:41.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056402+0000", 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1399999999999997e-07 2024-09-16T13:17:41.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056405+0000", 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6520000000000002e-06 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056411+0000", 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2009999999999996e-06 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056415+0000", 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.211e-06 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075482+0000", 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019067163000000002 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075546+0000", 2024-09-16T13:17:41.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3516000000000006e-05 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075844+0000", 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029810999999999998 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077784+0000", 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019399350000000001 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077823+0000", 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9524000000000003e-05 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077917+0000", 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4090000000000002e-05 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077959+0000", 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1615000000000001e-05 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8373 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.316405355000001, 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022897421000000001, 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.407 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3200000000000005e-07 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056427+0000", 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-06 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056433+0000", 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056437+0000", 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:17:41.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076470+0000", 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020032239 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076524+0000", 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4475000000000003e-05 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076996+0000", 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000471905 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.078749+0000", 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001752539 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079217+0000", 2024-09-16T13:17:41.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046819400000000001 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079276+0000", 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9407000000000002e-05 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079319+0000", 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:17:41.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8398 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.287041562999999, 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017439282, 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8398 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:41.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085788+0000", 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4719999999999998e-06 2024-09-16T13:17:41.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085793+0000", 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6179999999999997e-06 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085796+0000", 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6680000000000001e-06 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099514+0000", 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013717895000000001 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099873+0000", 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035970799999999998 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100269+0000", 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000395382 2024-09-16T13:17:41.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103068+0000", 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027992680000000002 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103114+0000", 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6189000000000001e-05 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103178+0000", 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3845999999999998e-05 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103225+0000", 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6888999999999998e-05 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8399 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.287000085999999, 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017551573000000001, 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.421 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8399 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085828+0000", 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085830+0000", 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8609999999999998e-06 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085835+0000", 2024-09-16T13:17:41.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.437e-06 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085837+0000", 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.571e-06 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100950+0000", 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015112537000000001 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101047+0000", 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6945000000000001e-05 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101468+0000", 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000420794 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103315+0000", 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018474769999999999 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103321+0000", 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103356+0000", 2024-09-16T13:17:41.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4764000000000001e-05 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103379+0000", 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3102999999999999e-05 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8400 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.427 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.286987185000001, 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017960752, 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8400 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.53e-07 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085842+0000", 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2620000000000001e-06 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085847+0000", 2024-09-16T13:17:41.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2119999999999997e-06 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085849+0000", 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.204e-06 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101995+0000", 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016146384999999999 2024-09-16T13:17:41.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102100+0000", 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010482199999999999 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102466+0000", 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000365731 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103389+0000", 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092367199999999995 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103702+0000", 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031220400000000001 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103754+0000", 2024-09-16T13:17:41.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2318999999999998e-05 2024-09-16T13:17:41.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103801+0000", 2024-09-16T13:17:41.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6687999999999998e-05 2024-09-16T13:17:41.434 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.434 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.434 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8539 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.12885563, 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018486425000000001, 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8539 2024-09-16T13:17:41.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0500000000000003e-07 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243975+0000", 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800000000000002e-06 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243982+0000", 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243986+0000", 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4100000000000001e-06 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258523+0000", 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014536604999999999 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258678+0000", 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155341 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.259218+0000", 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053930600000000003 2024-09-16T13:17:41.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262325+0000", 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031076799999999998 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262367+0000", 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1480000000000003e-05 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262417+0000", 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0555e-05 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262458+0000", 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0556999999999998e-05 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8540 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.128836208999999, 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018906683000000001, 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.442 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8540 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243992+0000", 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.75e-07 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243995+0000", 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1219999999999999e-06 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244001+0000", 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5010000000000001e-06 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244006+0000", 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999996e-06 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260137+0000", 2024-09-16T13:17:41.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016131 2024-09-16T13:17:41.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260200+0000", 2024-09-16T13:17:41.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3782999999999996e-05 2024-09-16T13:17:41.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260598+0000", 2024-09-16T13:17:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039746600000000002 2024-09-16T13:17:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262331+0000", 2024-09-16T13:17:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017327849999999999 2024-09-16T13:17:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262786+0000", 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045546300000000003 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262851+0000", 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-05 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262898+0000", 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7098000000000002e-05 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8541 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.128816986, 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019366889000000002, 2024-09-16T13:17:41.449 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8541 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244011+0000", 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244014+0000", 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.79e-06 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244020+0000", 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6370000000000002e-06 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244025+0000", 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3359999999999997e-06 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261122+0000", 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017097161 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261228+0000", 2024-09-16T13:17:41.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105989 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261475+0000", 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024757000000000002 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263022+0000", 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015466760000000001 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263300+0000", 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027828100000000001 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263344+0000", 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4008999999999998e-05 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263377+0000", 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2832e-05 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11776 3.1a 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.855413979000001, 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027432719000000001, 2024-09-16T13:17:41.456 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11776 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517415+0000", 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5239999999999999e-06 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526328+0000", 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089138059999999998 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526362+0000", 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3775000000000001e-05 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526379+0000", 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6835000000000001e-05 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526467+0000", 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7703999999999999e-05 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526653+0000", 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186379 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:41.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.527624+0000", 2024-09-16T13:17:41.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097080400000000002 2024-09-16T13:17:41.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543313+0000", 2024-09-16T13:17:41.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015689456000000001 2024-09-16T13:17:41.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544696+0000", 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013822979999999999 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544784+0000", 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8419999999999997e-05 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544846+0000", 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1717999999999994e-05 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:11778 3.1 e40/23 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head v 40'57, mlcod=40'55)", 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.848929951999999, 2024-09-16T13:17:41.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01745787, 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:41.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523898+0000", 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-06 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533067+0000", 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091691120000000001 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533101+0000", 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3733999999999999e-05 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533115+0000", 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3410999999999999e-05 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533194+0000", 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9827000000000007e-05 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533223+0000", 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8937000000000001e-05 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541297+0000", 2024-09-16T13:17:41.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080739850000000005 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541355+0000", 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7708999999999999e-05 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11777 3.1a 3:59cb7a3f:::rbd_object_map.112d21cb8818:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:41.468 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.846425041, 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018649718999999999, 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11777 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526404+0000", 2024-09-16T13:17:41.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.654e-06 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526409+0000", 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.882e-06 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526424+0000", 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5328999999999998e-05 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526428+0000", 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6219999999999998e-06 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542209+0000", 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015781765999999999 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542334+0000", 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125148 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542804+0000", 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046944899999999998 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543732+0000", 2024-09-16T13:17:41.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092824699999999999 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544929+0000", 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001196522 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545004+0000", 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4962999999999994e-05 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545052+0000", 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8136999999999997e-05 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11779 3.9 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.825903458999999, 2024-09-16T13:17:41.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020806181999999999, 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11779 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546926+0000", 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9769999999999999e-06 2024-09-16T13:17:41.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552330+0000", 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054043980000000004 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552353+0000", 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2875999999999999e-05 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552362+0000", 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7299999999999994e-06 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552447+0000", 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5620999999999999e-05 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552710+0000", 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026273300000000002 2024-09-16T13:17:41.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.553446+0000", 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073563299999999997 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.559002+0000", 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055564580000000002 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567615+0000", 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086126680000000004 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567689+0000", 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3839999999999998e-05 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567730+0000", 2024-09-16T13:17:41.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1248000000000002e-05 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13736 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.074832281999999, 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018229967999999999, 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.482 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13736 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2e-07 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297998+0000", 2024-09-16T13:17:41.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.824e-06 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298004+0000", 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8730000000000004e-06 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298008+0000", 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312651+0000", 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014643351000000001 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312740+0000", 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8466000000000003e-05 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313081+0000", 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034067899999999998 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315982+0000", 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002900943 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316051+0000", 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9029999999999995e-05 2024-09-16T13:17:41.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316101+0000", 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0680000000000003e-05 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316225+0000", 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012359099999999999 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.488 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.489 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13737 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:41.489 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:41.489 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.074815162, 2024-09-16T13:17:41.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018381208999999999, 2024-09-16T13:17:41.489 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.489 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13737 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298015+0000", 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6869999999999999e-06 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298021+0000", 2024-09-16T13:17:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7799999999999997e-06 2024-09-16T13:17:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298027+0000", 2024-09-16T13:17:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.071e-06 2024-09-16T13:17:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313829+0000", 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015801696 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314019+0000", 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018995700000000001 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:41.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314379+0000", 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036036500000000002 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315988+0000", 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016084949999999999 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316290+0000", 2024-09-16T13:17:41.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000302732 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316324+0000", 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3937999999999999e-05 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316393+0000", 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8966000000000003e-05 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.495 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13738 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.07479618, 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021400311000000002, 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13738 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298032+0000", 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7400000000000003e-07 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298034+0000", 2024-09-16T13:17:41.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5770000000000001e-06 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298040+0000", 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7010000000000001e-06 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298044+0000", 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2069999999999997e-06 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315033+0000", 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016988970999999999 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315106+0000", 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2723000000000002e-05 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:41.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315428+0000", 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032191099999999997 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319249+0000", 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003821166 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319313+0000", 2024-09-16T13:17:41.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319347+0000", 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4591999999999997e-05 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319431+0000", 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3893000000000005e-05 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.502 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13739 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.074778668, 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021506517999999999, 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13739 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298049+0000", 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298052+0000", 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.819e-06 2024-09-16T13:17:41.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298058+0000", 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6960000000000002e-06 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298061+0000", 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7560000000000001e-06 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316465+0000", 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018403744 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316747+0000", 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028175200000000002 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.317694+0000", 2024-09-16T13:17:41.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000947115 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319304+0000", 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016102029999999999 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319446+0000", 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142008 2024-09-16T13:17:41.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319490+0000", 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3690999999999999e-05 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319555+0000", 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5191000000000007e-05 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.509 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13740 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.074725184999998, 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023881870999999999, 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13740 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:41.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298103+0000", 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298106+0000", 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1559999999999999e-06 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298120+0000", 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4474999999999999e-05 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298124+0000", 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1699999999999999e-06 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319635+0000", 2024-09-16T13:17:41.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021510129999999999 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319762+0000", 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012746400000000001 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.320225+0000", 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046264800000000002 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321629+0000", 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00140383 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321682+0000", 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3720000000000001e-05 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321757+0000", 2024-09-16T13:17:41.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:17:41.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321984+0000", 2024-09-16T13:17:41.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022684199999999999 2024-09-16T13:17:41.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.516 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.516 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.517 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:41.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:41.520 INFO:teuthology.orchestra.run.smithi098.stdout:3154 write 0xc410838 thru 0xc41fbd1 (0xf39a bytes) 2024-09-16T13:17:41.520 INFO:teuthology.orchestra.run.smithi098.stdout:3155 write 0x619b5dd thru 0x61a6e4f (0xb873 bytes) 2024-09-16T13:17:41.520 INFO:teuthology.orchestra.run.smithi098.stdout:3156 write 0x9d05134 thru 0x9d11ee8 (0xcdb5 bytes) 2024-09-16T13:17:41.520 INFO:teuthology.orchestra.run.smithi098.stdout:3158 writesame 0x1ef9336 thru 0x1efcb9c data_size 0x1(0x3867 bytes) 2024-09-16T13:17:41.520 INFO:teuthology.orchestra.run.smithi098.stdout:3163 read 0x430e02d thru 0x431154b (0x351f bytes) 2024-09-16T13:17:41.520 INFO:teuthology.orchestra.run.smithi098.stdout:3165 read 0x9530246 thru 0x95302fd (0xb8 bytes) 2024-09-16T13:17:41.520 INFO:teuthology.orchestra.run.smithi098.stdout:3167 writesame 0x4733ba0 thru 0x473bb45 data_size 0x1(0x7fa6 bytes) 2024-09-16T13:17:41.520 INFO:teuthology.orchestra.run.smithi098.stdout:3169 trunc from 0xc41fbd2 to 0xeb87317 2024-09-16T13:17:41.520 INFO:teuthology.orchestra.run.smithi098.stdout:3170 writesame 0xe89b724 thru 0xe89fb32 data_size 0x13(0x440f bytes) 2024-09-16T13:17:41.521 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:41.508+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:41.528 INFO:teuthology.orchestra.run.smithi098.stdout:3171 trunc from 0xeb87317 to 0xae66a39 2024-09-16T13:17:41.569 INFO:teuthology.orchestra.run.smithi098.stdout:3172 write 0xbb9efb7 thru 0xbba604f (0x7099 bytes) 2024-09-16T13:17:41.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:17:41.576 INFO:teuthology.orchestra.run.smithi098.stdout:3173 read 0xc0d93c thru 0xc187b6 (0xae7b bytes) 2024-09-16T13:17:41.577 INFO:teuthology.orchestra.run.smithi098.stdout:3174 read 0x4c4dd1d thru 0x4c5807a (0xa35e bytes) 2024-09-16T13:17:41.577 INFO:teuthology.orchestra.run.smithi098.stdout:3175 punch from 0xb6e8fe1 to 0xb6f3073, (0xa092 bytes) 2024-09-16T13:17:41.577 INFO:teuthology.orchestra.run.smithi098.stdout:3176 write 0x1d46eeb thru 0x1d490e0 (0x21f6 bytes) 2024-09-16T13:17:41.581 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:41.581 INFO:teuthology.orchestra.run.smithi098.stdout:3180 read 0x7d9d2d8 thru 0x7da5005 (0x7d2e bytes) 2024-09-16T13:17:41.582 INFO:teuthology.orchestra.run.smithi098.stdout:3181 punch from 0xb7caa91 to 0xb7d301d, (0x858c bytes) 2024-09-16T13:17:41.582 INFO:teuthology.orchestra.run.smithi098.stdout:3182 writesame 0x3c2a47b thru 0x3c358ef data_size 0x57(0xb475 bytes) 2024-09-16T13:17:41.589 INFO:teuthology.orchestra.run.smithi098.stdout:3183 read 0x858a2d7 thru 0x858b1bf (0xee9 bytes) 2024-09-16T13:17:41.590 INFO:teuthology.orchestra.run.smithi098.stdout:3184 read 0xa3cffc0 thru 0xa3df883 (0xf8c4 bytes) 2024-09-16T13:17:41.590 INFO:teuthology.orchestra.run.smithi098.stdout:3186 read 0x4d8cd56 thru 0x4d93121 (0x63cc bytes) 2024-09-16T13:17:41.592 INFO:teuthology.orchestra.run.smithi098.stdout:3188 trunc from 0xbba6050 to 0xb20e883 2024-09-16T13:17:41.607 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:41.607 INFO:teuthology.orchestra.run.smithi098.stdout:3190 punch from 0x2c975ee to 0x2ca2c80, (0xb692 bytes) 2024-09-16T13:17:41.610 INFO:teuthology.orchestra.run.smithi098.stdout:3191 trunc from 0xb20e883 to 0x9f9dc56 2024-09-16T13:17:41.619 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:41.608+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:41.625 INFO:teuthology.orchestra.run.smithi098.stdout:3192 write 0x203cc7f thru 0x2044690 (0x7a12 bytes) 2024-09-16T13:17:41.628 INFO:teuthology.orchestra.run.smithi098.stdout:3193 read 0x57bd129 thru 0x57bf7dc (0x26b4 bytes) 2024-09-16T13:17:41.629 INFO:teuthology.orchestra.run.smithi098.stdout:3194 punch from 0xeb20e6 to 0xeb35de, (0x14f8 bytes) 2024-09-16T13:17:41.646 INFO:teuthology.orchestra.run.smithi098.stdout:3195 writesame 0xbf327f7 thru 0xbf3296c data_size 0xb(0x176 bytes) 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi098.stdout:3196 punch from 0x821cf2a to 0x8222ae4, (0x5bba bytes) 2024-09-16T13:17:41.649 INFO:teuthology.orchestra.run.smithi098.stdout:3198 clone 18 order 24 su 1048576 sc 6 2024-09-16T13:17:41.675 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:41.675 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:41.675 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:41.676 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:41.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:41.719 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:41.716+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:41.779 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:41.779 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:41.779 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:41.779 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:41.779 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:41.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:41.833 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:41.827+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:41.892 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:41.892 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:41.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:41.892 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2522 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3473408~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861715048000001, 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013780114, 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2522 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.893 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024641+0000", 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8999999999999998e-07 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024642+0000", 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3230000000000001e-06 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024646+0000", 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024648+0000", 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4849999999999999e-06 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038251+0000", 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013602822000000001 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038335+0000", 2024-09-16T13:17:41.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4215999999999995e-05 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038421+0000", 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.543e-05 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2523 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861702102999999, 2024-09-16T13:17:41.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014015147, 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2523 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024656+0000", 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.131e-06 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024660+0000", 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6990000000000001e-06 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024663+0000", 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0290000000000001e-06 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038456+0000", 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013793481 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038549+0000", 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2608000000000005e-05 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038669+0000", 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011982800000000001 2024-09-16T13:17:41.901 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2524 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861688528999998, 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014381392, 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.902 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2524 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024668+0000", 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8400000000000003e-07 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024670+0000", 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.849e-06 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024675+0000", 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.451e-06 2024-09-16T13:17:41.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024678+0000", 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8739999999999999e-06 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038723+0000", 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014044562999999999 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038811+0000", 2024-09-16T13:17:41.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039049+0000", 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023761100000000001 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2525 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 557056~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:41.906 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861654182000002, 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014602618, 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2525 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:41.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024705+0000", 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4650000000000001e-06 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024711+0000", 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024716+0000", 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1710000000000004e-06 2024-09-16T13:17:41.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039096+0000", 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014380184000000001 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039187+0000", 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1065999999999999e-05 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039304+0000", 2024-09-16T13:17:41.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117333 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2526 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1507328~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861636013000002, 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014823846, 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2526 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024722+0000", 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024729+0000", 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024733+0000", 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039337+0000", 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014604026000000001 2024-09-16T13:17:41.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039427+0000", 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0437000000000002e-05 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039544+0000", 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116234 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2527 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1605632~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861618249999999, 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015041116, 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2527 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024738+0000", 2024-09-16T13:17:41.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024746+0000", 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2130000000000008e-06 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024753+0000", 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024758+0000", 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7670000000000003e-06 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014820533 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039670+0000", 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1825999999999998e-05 2024-09-16T13:17:41.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039779+0000", 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108424 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2528 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861592811999998, 2024-09-16T13:17:41.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015245242000000001, 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2528 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024765+0000", 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1040000000000001e-06 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024772+0000", 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024776+0000", 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039810+0000", 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015034194000000001 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039897+0000", 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6864e-05 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040008+0000", 2024-09-16T13:17:41.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110802 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2529 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2555904~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861573692999997, 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015471574, 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.925 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2529 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024785+0000", 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5969999999999999e-06 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024800+0000", 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4681999999999999e-05 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024806+0000", 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040039+0000", 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015233007999999999 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040143+0000", 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010322799999999999 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040254+0000", 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011092899999999999 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.929 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2530 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861543413, 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015667606000000001, 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2530 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024813+0000", 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024815+0000", 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5160000000000001e-06 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024827+0000", 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1915e-05 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024834+0000", 2024-09-16T13:17:41.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5180000000000001e-06 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040286+0000", 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015452565999999999 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040375+0000", 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040480+0000", 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105388 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2531 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3538944~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861516197999997, 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015867612999999999, 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2531 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024843+0000", 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6960000000000001e-06 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024856+0000", 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-05 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024863+0000", 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6069999999999996e-06 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040513+0000", 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650662999999999 2024-09-16T13:17:41.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040599+0000", 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5439999999999995e-05 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040707+0000", 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108461 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2532 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861485692999999, 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061912000000001, 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2532 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.939 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024871+0000", 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024874+0000", 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024881+0000", 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0570000000000004e-06 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024886+0000", 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040742+0000", 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015856808 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040824+0000", 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:17:41.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040932+0000", 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108378 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2533 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861465168000002, 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016272282999999998, 2024-09-16T13:17:41.943 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2533 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:41.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:41.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:17:41.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024893+0000", 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419999999999999e-06 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024900+0000", 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024903+0000", 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040965+0000", 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061565999999999 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041052+0000", 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7213000000000007e-05 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041163+0000", 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011066899999999999 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2534 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3604480~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861446637, 2024-09-16T13:17:41.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016482746999999999, 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2534 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024910+0000", 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1299999999999999e-07 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024912+0000", 2024-09-16T13:17:41.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024920+0000", 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024924+0000", 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.036e-06 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041197+0000", 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016273242 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041282+0000", 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5013999999999996e-05 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041392+0000", 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109567 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.952 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2535 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861425975000003, 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016691902000000002, 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.953 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2535 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7599999999999997e-07 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024932+0000", 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024949+0000", 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7053000000000001e-05 2024-09-16T13:17:41.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024961+0000", 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041425+0000", 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016463851000000002 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041515+0000", 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9549999999999995e-05 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041622+0000", 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107132 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2536 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1638400~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.957 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861388169000001, 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016880869999999999, 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2536 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024971+0000", 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.615e-06 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024988+0000", 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6912999999999999e-05 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024993+0000", 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041653+0000", 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016660245000000001 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041743+0000", 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0099999999999995e-05 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041848+0000", 2024-09-16T13:17:41.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105404 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2537 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1671168~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861357415999997, 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017075346000000002, 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2537 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024999+0000", 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025002+0000", 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025009+0000", 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-09-16T13:17:41.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025013+0000", 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1840000000000001e-06 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041886+0000", 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016873249999999999 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041971+0000", 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4487000000000006e-05 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042074+0000", 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102847 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2538 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2621440~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.966 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861337245000001, 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017314057000000001, 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2538 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:41.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025019+0000", 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025021+0000", 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025033+0000", 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1985e-05 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025043+0000", 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3810000000000005e-06 2024-09-16T13:17:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042107+0000", 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017064471000000001 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042203+0000", 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5810999999999995e-05 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042333+0000", 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012956099999999999 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2539 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.861309517000002, 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017500180000000001, 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2539 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025047+0000", 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025053+0000", 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025058+0000", 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025061+0000", 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.072e-06 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042365+0000", 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017304885999999998 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042458+0000", 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2991000000000005e-05 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042546+0000", 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7904000000000004e-05 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:22786 3.17 3:edd0f01e:::rbd_object_map.112db5030c67:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:41.317386+0000", 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.568970058, 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013576861000000001, 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 22786 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317386+0000", 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317386+0000", 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317387+0000", 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1200000000000001e-06 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317394+0000", 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9399999999999996e-06 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317410+0000", 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6419999999999999e-05 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317420+0000", 2024-09-16T13:17:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7799999999999995e-06 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.327271+0000", 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0098510890000000004 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.327429+0000", 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015777799999999999 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.328025+0000", 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00059603800000000001 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.330865+0000", 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028399380000000002 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.330871+0000", 2024-09-16T13:17:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6259999999999997e-06 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.330923+0000", 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1193000000000003e-05 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.330963+0000", 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9938999999999998e-05 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:22787 3.17 3:edd0f01e:::rbd_object_map.112db5030c67:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.56892493499999996, 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015269972, 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 22787 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317432+0000", 2024-09-16T13:17:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.127e-06 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317439+0000", 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0500000000000003e-06 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317458+0000", 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9326000000000001e-05 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317467+0000", 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0669999999999998e-06 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329243+0000", 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011775551 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329293+0000", 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9709000000000001e-05 2024-09-16T13:17:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329505+0000", 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021237200000000001 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.330969+0000", 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014635290000000001 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332590+0000", 2024-09-16T13:17:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001621485 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332655+0000", 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4670000000000003e-05 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332701+0000", 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6085999999999999e-05 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:41.988 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:41.988 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:41.988 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:41.988 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-16T13:17:41.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-09-16T13:17:41.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T13:17:41.888+0000 7f08796f3700 0 --2- 172.21.15.98:0/3103115293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f087406aea0 0x7f08741328e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T13:17:41.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:41.991 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:41.983+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:42.090 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:42.083+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:42.100 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:42.100 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:42.100 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:42.100 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:42.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:42.191 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:42.183+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:42.257 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:42.257 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:42.257 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:42.257 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:42.257 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:42.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:42.288 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-09-16T13:17:42.280+0000 7f1b776ba700 -1 mon.a@0(leader).osd e57 definitely_dead 0 2024-09-16T13:17:42.292 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:42.283+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:42.362 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:42.362 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:42.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:42.362 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:42.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.362 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11487 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 430080~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.362 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:42.362 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.870912814, 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014286111000000001, 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11487 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486860+0000", 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.04e-06 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486866+0000", 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0629999999999996e-06 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486881+0000", 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5023e-05 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486889+0000", 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8149999999999995e-06 2024-09-16T13:17:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.500906+0000", 2024-09-16T13:17:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014016416 2024-09-16T13:17:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501008+0000", 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102243 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501146+0000", 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013751099999999999 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11535 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.870229841, 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014337727999999999, 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11535 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487543+0000", 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023e-06 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487551+0000", 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0309999999999998e-06 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487566+0000", 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5000999999999999e-05 2024-09-16T13:17:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487578+0000", 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1066e-05 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501655+0000", 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014077151 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501747+0000", 2024-09-16T13:17:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2670999999999994e-05 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501880+0000", 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013278500000000001 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11541 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 413696~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.870185290999999, 2024-09-16T13:17:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014490381, 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11541 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487588+0000", 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487592+0000", 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4920000000000004e-06 2024-09-16T13:17:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487607+0000", 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4926000000000001e-05 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487616+0000", 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6829999999999997e-06 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501906+0000", 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0142898 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501959+0000", 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2985000000000002e-05 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502077+0000", 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118495 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11542 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 393216~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.870146766000001, 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014676434, 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11542 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487626+0000", 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.034e-06 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487631+0000", 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7659999999999998e-06 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487646+0000", 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4786e-05 2024-09-16T13:17:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487655+0000", 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6489999999999997e-06 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502110+0000", 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014455749 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502209+0000", 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8608000000000002e-05 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502302+0000", 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2842000000000003e-05 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11544 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 466944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.870105805000001, 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014846796000000001, 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11544 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487667+0000", 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.569999999999999e-07 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487672+0000", 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2459999999999997e-06 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487686+0000", 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4847e-05 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487695+0000", 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9109999999999999e-06 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502337+0000", 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014641861000000001 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502440+0000", 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010282599999999999 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502513+0000", 2024-09-16T13:17:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3147999999999998e-05 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11547 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.870066678000001, 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015113645, 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11547 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487707+0000", 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.015e-06 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487711+0000", 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8289999999999998e-06 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487727+0000", 2024-09-16T13:17:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5497000000000001e-05 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487736+0000", 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1409999999999994e-06 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502557+0000", 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014821212 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502675+0000", 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117741 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502819+0000", 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014421 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11549 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 413696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.870026255999999, 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016244525999999999, 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11549 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487747+0000", 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.094e-06 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487752+0000", 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.758e-06 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487770+0000", 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8502999999999999e-05 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487779+0000", 2024-09-16T13:17:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1579999999999994e-06 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502868+0000", 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015088811000000001 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502916+0000", 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8109000000000003e-05 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503990+0000", 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010740929999999999 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11578 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 409600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.869975814, 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015410814, 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11578 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487798+0000", 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.646e-06 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487804+0000", 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:17:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487821+0000", 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7677999999999999e-05 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487830+0000", 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9090000000000006e-06 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503033+0000", 2024-09-16T13:17:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015202337999999999 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503123+0000", 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0414000000000005e-05 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503207+0000", 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4172000000000007e-05 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11580 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.869932177999999, 2024-09-16T13:17:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015544980999999999, 2024-09-16T13:17:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11580 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487841+0000", 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487846+0000", 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3919999999999996e-06 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487861+0000", 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4946000000000001e-05 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487869+0000", 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8750000000000006e-06 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503240+0000", 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015370511999999999 2024-09-16T13:17:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503334+0000", 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4265000000000006e-05 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503385+0000", 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0773999999999999e-05 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11617 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 430080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.868612479999999, 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376203000000001, 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11617 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489162+0000", 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3089999999999998e-06 2024-09-16T13:17:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489171+0000", 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3430000000000002e-06 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489198+0000", 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6449999999999999e-05 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489216+0000", 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8057999999999999e-05 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503402+0000", 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014186196999999999 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503454+0000", 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1989e-05 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503536+0000", 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1856999999999999e-05 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11624 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 503808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.868409392, 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376676, 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11624 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489365+0000", 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.795e-06 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489372+0000", 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4030000000000001e-06 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489392+0000", 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0129e-05 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489404+0000", 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1555e-05 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503558+0000", 2024-09-16T13:17:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014154771999999999 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503644+0000", 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5691000000000003e-05 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503740+0000", 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5330999999999999e-05 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11630 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.868356055, 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014893777, 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11630 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489417+0000", 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6800000000000009e-07 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489435+0000", 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7818000000000001e-05 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489450+0000", 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5072000000000001e-05 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489459+0000", 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3009999999999995e-06 2024-09-16T13:17:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503787+0000", 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01432719 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503889+0000", 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102604 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504310+0000", 2024-09-16T13:17:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042082399999999998 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11632 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.868302942, 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01524066, 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11632 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489470+0000", 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.78e-07 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489475+0000", 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9400000000000001e-06 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489488+0000", 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3176999999999999e-05 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489497+0000", 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4919999999999993e-06 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504364+0000", 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014866846 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504485+0000", 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012142999999999999 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504710+0000", 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000224897 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11639 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.868261171, 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015375941000000001, 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11639 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489512+0000", 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489517+0000", 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.938e-06 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489532+0000", 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.47e-05 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489540+0000", 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2949999999999994e-06 2024-09-16T13:17:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504765+0000", 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015225449 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504832+0000", 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6929999999999998e-05 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504887+0000", 2024-09-16T13:17:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4681e-05 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11641 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.868214661, 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015522312999999999, 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11641 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489559+0000", 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.066e-06 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489564+0000", 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9370000000000003e-06 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489579+0000", 2024-09-16T13:17:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5466000000000001e-05 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489588+0000", 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504925+0000", 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015337353 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505025+0000", 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9770000000000002e-05 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505080+0000", 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4792e-05 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11667 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 16384~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.867853891999999, 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015353510000000001, 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11667 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489920+0000", 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.84e-06 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489928+0000", 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6969999999999993e-06 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489953+0000", 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5154999999999999e-05 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489965+0000", 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2292e-05 2024-09-16T13:17:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505115+0000", 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015149793999999999 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505184+0000", 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8858999999999999e-05 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505272+0000", 2024-09-16T13:17:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7873000000000004e-05 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11672 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 36864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.867793487, 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015615053, 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11672 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489980+0000", 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489985+0000", 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490000+0000", 2024-09-16T13:17:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490010+0000", 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.183e-06 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505335+0000", 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015325167000000001 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505471+0000", 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013581699999999999 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505594+0000", 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123168 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11676 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.867752743, 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015779395000000002, 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11676 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490021+0000", 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.575e-06 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490026+0000", 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5170000000000001e-06 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490041+0000", 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4955e-05 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490049+0000", 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5520000000000005e-06 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505647+0000", 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015597489000000001 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505743+0000", 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6908000000000001e-05 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505799+0000", 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5399000000000003e-05 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11678 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.867713469999998, 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015992643000000001, 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11678 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490060+0000", 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3400000000000001e-06 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490065+0000", 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4830000000000001e-06 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490079+0000", 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4854999999999999e-05 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490088+0000", 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1119999999999996e-06 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505849+0000", 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015761718000000001 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505966+0000", 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011645399999999999 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506051+0000", 2024-09-16T13:17:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5680999999999995e-05 2024-09-16T13:17:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11707 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 53248~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.867494341, 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016106869999999999, 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11707 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490280+0000", 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7659999999999999e-06 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490287+0000", 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.503e-06 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490307+0000", 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9989000000000001e-05 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490318+0000", 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0967e-05 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506171+0000", 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015853367 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506265+0000", 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3190999999999996e-05 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506385+0000", 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120087 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.455 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:42.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:42.457 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:42.447+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:42.557 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:42.547+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:42.572 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:17:42.576 INFO:teuthology.orchestra.run.smithi098.stderr:marked in osd.0. 2024-09-16T13:17:42.587 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-16T13:17:42.603 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:42.603 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:42.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:42.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:42.658 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:42.648+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:42.708 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:42.709 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:42.709 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:42.709 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:42.709 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:42.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:42.759 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:42.748+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:42.766 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:42.787 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:42.815 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:42.825 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:42.825 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:42.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:42.825 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.793743669999998, 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:17:42.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:17:42.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:17:42.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:17:42.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:17:42.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:17:42.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:17:42.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:42.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:17:42.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:17:42.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:42.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:17:42.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:17:42.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:42.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:17:42.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:17:42.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:17:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:17:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:42.833 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.793596264000001, 2024-09-16T13:17:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:17:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:17:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:42.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:42.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:17:42.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:17:42.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:17:42.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:17:42.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:17:42.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:17:42.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:17:42.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:17:42.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:17:42.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:17:42.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:42.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:17:42.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:42.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:17:42.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:17:42.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:42.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.793568921000002, 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:42.841 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:42.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:17:42.846 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.234893658999994, 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:42.847 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:17:42.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:17:42.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:42.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.852 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.234806066999994, 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:42.853 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:17:42.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:42.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:17:42.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:17:42.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:42.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:17:42.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:17:42.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.234636144999996, 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.859 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:17:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:17:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:17:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:17:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:17:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:17:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:17:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:17:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:17:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:17:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:17:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:17:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:17:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:17:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:17:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:17:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:17:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:17:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.866 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2259 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3014656~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083307568999999, 2024-09-16T13:17:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016650893999999999, 2024-09-16T13:17:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.867 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2259 2024-09-16T13:17:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:42.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:42.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735831+0000", 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9530000000000005e-06 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735836+0000", 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7450000000000003e-06 2024-09-16T13:17:42.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735839+0000", 2024-09-16T13:17:42.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.915e-06 2024-09-16T13:17:42.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752296+0000", 2024-09-16T13:17:42.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016457241000000001 2024-09-16T13:17:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752360+0000", 2024-09-16T13:17:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4108000000000004e-05 2024-09-16T13:17:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752473+0000", 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112305 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2260 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083287648000002, 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016799073000000001, 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.873 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2260 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:42.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735843+0000", 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3480000000000001e-06 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735847+0000", 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5990000000000002e-06 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735849+0000", 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5859999999999999e-06 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752509+0000", 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016659377 2024-09-16T13:17:42.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752559+0000", 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9774000000000002e-05 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752641+0000", 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2057000000000004e-05 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.877 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2261 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083275458999999, 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016960698, 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2261 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.03e-07 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735855+0000", 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735860+0000", 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1110000000000001e-06 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735862+0000", 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752695+0000", 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01683343 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752744+0000", 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8414000000000001e-05 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752814+0000", 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0623000000000005e-05 2024-09-16T13:17:42.881 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2262 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1048576~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083264845000002, 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017224107999999998, 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.882 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2262 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735865+0000", 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.91e-07 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735866+0000", 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1480000000000001e-06 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735870+0000", 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735877+0000", 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4370000000000001e-06 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752871+0000", 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016994061000000001 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752962+0000", 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0470000000000006e-05 2024-09-16T13:17:42.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753089+0000", 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126642 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2263 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083249385999999, 2024-09-16T13:17:42.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017392088999999999, 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2263 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-07 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735881+0000", 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-09-16T13:17:42.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735885+0000", 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4180000000000001e-06 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735887+0000", 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-06 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753116+0000", 2024-09-16T13:17:42.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017228963 2024-09-16T13:17:42.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753184+0000", 2024-09-16T13:17:42.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7374999999999998e-05 2024-09-16T13:17:42.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753272+0000", 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8323000000000001e-05 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2264 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2064384~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083239534999997, 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017630541999999999, 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2264 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:42.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:42.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:42.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735891+0000", 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735895+0000", 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735897+0000", 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4669999999999998e-06 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753316+0000", 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017418685 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753403+0000", 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7590000000000007e-05 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753520+0000", 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116863 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2265 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1081344~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083229652999997, 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017739263000000002, 2024-09-16T13:17:42.896 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.897 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.897 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.897 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.897 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.897 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2265 2024-09-16T13:17:42.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.897 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:42.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:42.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735901+0000", 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735904+0000", 2024-09-16T13:17:42.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.304e-06 2024-09-16T13:17:42.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735907+0000", 2024-09-16T13:17:42.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3599999999999999e-06 2024-09-16T13:17:42.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753534+0000", 2024-09-16T13:17:42.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017627446000000001 2024-09-16T13:17:42.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753576+0000", 2024-09-16T13:17:42.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1474999999999998e-05 2024-09-16T13:17:42.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753639+0000", 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3188999999999994e-05 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2266 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2097152~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083220420000004, 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017970893000000002, 2024-09-16T13:17:42.902 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2266 2024-09-16T13:17:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:42.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:42.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:42.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:42.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735914+0000", 2024-09-16T13:17:42.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1699999999999996e-06 2024-09-16T13:17:42.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735918+0000", 2024-09-16T13:17:42.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6859999999999999e-06 2024-09-16T13:17:42.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735920+0000", 2024-09-16T13:17:42.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4870000000000001e-06 2024-09-16T13:17:42.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753685+0000", 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017764582000000001 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753771+0000", 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6259000000000002e-05 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753880+0000", 2024-09-16T13:17:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010844300000000001 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2267 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083206519000001, 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018110804000000001, 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2267 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735924+0000", 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735928+0000", 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.416e-06 2024-09-16T13:17:42.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735930+0000", 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753898+0000", 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017967852999999999 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753955+0000", 2024-09-16T13:17:42.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6662000000000001e-05 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754034+0000", 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8904000000000002e-05 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2268 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3080192~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083197034000001, 2024-09-16T13:17:42.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018252866, 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2268 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735934+0000", 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0950000000000001e-06 2024-09-16T13:17:42.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735938+0000", 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735941+0000", 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8750000000000002e-06 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754071+0000", 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018129941 2024-09-16T13:17:42.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754120+0000", 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8359000000000002e-05 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754185+0000", 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5358999999999995e-05 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2269 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083185655000001, 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018533516, 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2269 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735945+0000", 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1149999999999999e-06 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735948+0000", 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5250000000000001e-06 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735951+0000", 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9909999999999998e-06 2024-09-16T13:17:42.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754270+0000", 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018318081999999999 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754325+0000", 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5648e-05 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754477+0000", 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015188199999999999 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2270 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1114112~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083175185999998, 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019103282999999999, 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2270 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:42.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735955+0000", 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735956+0000", 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.096e-06 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735959+0000", 2024-09-16T13:17:42.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5140000000000002e-06 2024-09-16T13:17:42.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735963+0000", 2024-09-16T13:17:42.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.053e-06 2024-09-16T13:17:42.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754822+0000", 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018858883 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754941+0000", 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118789 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755057+0000", 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116272 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2271 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1146880~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083153373999998, 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019332309999999998, 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.926 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.927 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.927 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.927 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.927 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2271 2024-09-16T13:17:42.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.927 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:42.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3500000000000002e-07 2024-09-16T13:17:42.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735982+0000", 2024-09-16T13:17:42.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:17:42.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735987+0000", 2024-09-16T13:17:42.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1930000000000003e-06 2024-09-16T13:17:42.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735993+0000", 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4659999999999996e-06 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755081+0000", 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019088187 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755194+0000", 2024-09-16T13:17:42.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011223299999999999 2024-09-16T13:17:42.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755308+0000", 2024-09-16T13:17:42.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114663 2024-09-16T13:17:42.931 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.931 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.931 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.932 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2272 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 163840~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:42.932 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:42.932 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.083131543999997, 2024-09-16T13:17:42.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019476291, 2024-09-16T13:17:42.932 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:42.932 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:42.932 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:42.932 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2272 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:42.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:42.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:42.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:42.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:42.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:42.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:42.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736003+0000", 2024-09-16T13:17:42.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2279999999999998e-06 2024-09-16T13:17:42.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:42.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736007+0000", 2024-09-16T13:17:42.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.506e-06 2024-09-16T13:17:42.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:42.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736011+0000", 2024-09-16T13:17:42.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9280000000000003e-06 2024-09-16T13:17:42.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:42.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755336+0000", 2024-09-16T13:17:42.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019325189999999999 2024-09-16T13:17:42.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:42.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755393+0000", 2024-09-16T13:17:42.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7293999999999997e-05 2024-09-16T13:17:42.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:42.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:42.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:42.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755474+0000", 2024-09-16T13:17:42.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0635000000000003e-05 2024-09-16T13:17:42.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.938 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:42.938 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:42.938 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:42.940 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:42.940 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:42.940 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:42.940 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:42.940 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:42.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:42.942 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:42.942 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:42.942 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:42.942 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:42.942 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:42.942 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:42.942 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:42.942 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:42.943 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:42.932+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:43.041 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:43.032+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:43.101 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:43.101 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:43.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:43.159 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:43.152+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:43.202 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:43.202 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:43.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:43.243 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:43.235+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:43.307 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:43.308 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:43.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:43.344 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:43.335+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:43.415 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:43.415 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:43.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:43.445 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:43.435+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:43.525 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:43.525 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:43.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:43.546 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:43.543+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:43.633 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:43.633 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:43.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:43.646 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:43.643+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:43.738 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:43.738 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:43.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:43.748 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:43.740+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:43.845 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:43.845 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:43.848 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:43.840+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:43.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:43.949 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:43.943+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:44.002 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:44.003 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:44.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:44.049 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:44.040+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:44.115 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:44.115 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:44.124 INFO:teuthology.orchestra.run.smithi098.stdout:flattening image image_client.0-clone17 2024-09-16T13:17:44.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:44.150 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:44.140+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:44.232 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:44.232 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:44.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:44.251 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:44.243+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:44.323 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #16, image image_client.0-clone16 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent17 2024-09-16T13:17:44.352 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:44.344+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:44.404 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:44.404 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:44.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:44.453 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:44.444+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:44.523 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:44.523 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:44.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:44.554 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:44.544+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:44.575 INFO:teuthology.orchestra.run.smithi098.stdout:3200 write 0x69aad3f thru 0x69adb78 (0x2e3a bytes) 2024-09-16T13:17:44.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:17:44.590 INFO:teuthology.orchestra.run.smithi098.stdout:3201 read 0xae4241f thru 0xae4d018 (0xabfa bytes) 2024-09-16T13:17:44.590 INFO:teuthology.orchestra.run.smithi098.stdout:3202 punch from 0x54fd132 to 0x550bf2e, (0xedfc bytes) 2024-09-16T13:17:44.590 INFO:teuthology.orchestra.run.smithi098.stdout:3203 write 0x7636a4e thru 0x763b226 (0x47d9 bytes) 2024-09-16T13:17:44.598 INFO:teuthology.orchestra.run.smithi098.stdout:3204 punch from 0x6a559cc to 0x6a5f1ef, (0x9823 bytes) 2024-09-16T13:17:44.598 INFO:teuthology.orchestra.run.smithi098.stdout:3205 writesame 0x9856589 thru 0x985684f data_size 0x3(0x2c7 bytes) 2024-09-16T13:17:44.604 INFO:teuthology.orchestra.run.smithi098.stdout:3206 writesame 0x5948de7 thru 0x594bd94 data_size 0x2(0x2fae bytes) 2024-09-16T13:17:44.615 INFO:teuthology.orchestra.run.smithi098.stdout:3207 write 0x5bbf17a thru 0x5bbfa5a (0x8e1 bytes) 2024-09-16T13:17:44.622 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:44.622 INFO:teuthology.orchestra.run.smithi098.stdout:3209 punch from 0x199088e to 0x199af94, (0xa706 bytes) 2024-09-16T13:17:44.622 INFO:teuthology.orchestra.run.smithi098.stdout:3210 writesame 0x14f3050 thru 0x1500425 data_size 0x37(0xd3d6 bytes) 2024-09-16T13:17:44.631 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:44.631 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:44.631 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:44.631 INFO:teuthology.orchestra.run.smithi098.stdout:3215 punch from 0x9676768 to 0x9684667, (0xdeff bytes) 2024-09-16T13:17:44.631 INFO:teuthology.orchestra.run.smithi098.stdout:3216 write 0x9bc30bb thru 0x9bd0a3f (0xd985 bytes) 2024-09-16T13:17:44.638 INFO:teuthology.orchestra.run.smithi098.stdout:3217 read 0xb1000b2 thru 0xb10d8ad (0xd7fc bytes) 2024-09-16T13:17:44.638 INFO:teuthology.orchestra.run.smithi098.stdout:3219 read 0x6defd1c thru 0x6df9390 (0x9675 bytes) 2024-09-16T13:17:44.639 INFO:teuthology.orchestra.run.smithi098.stdout:3221 punch from 0x84f8c54 to 0x84fad49, (0x20f5 bytes) 2024-09-16T13:17:44.641 INFO:teuthology.orchestra.run.smithi098.stdout:3222 trunc from 0xbf3296d to 0xd739107 2024-09-16T13:17:44.649 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:44.649 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:44.655 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:44.647+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:44.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:44.666 INFO:teuthology.orchestra.run.smithi098.stdout:3223 trunc from 0xd739107 to 0x4b1301e 2024-09-16T13:17:44.693 INFO:teuthology.orchestra.run.smithi098.stdout:3224 trunc from 0x4b1301e to 0x4a73da0 2024-09-16T13:17:44.706 INFO:teuthology.orchestra.run.smithi098.stdout:3225 read 0x1f17e6 thru 0x1fdd2d (0xc548 bytes) 2024-09-16T13:17:44.706 INFO:teuthology.orchestra.run.smithi098.stdout:3226 read 0x2a27d0 thru 0x2ac8a6 (0xa0d7 bytes) 2024-09-16T13:17:44.707 INFO:teuthology.orchestra.run.smithi098.stdout:3227 read 0x27e9b4e thru 0x27f33b9 (0x986c bytes) 2024-09-16T13:17:44.708 INFO:teuthology.orchestra.run.smithi098.stdout:3228 read 0x48b342b thru 0x48b8ece (0x5aa4 bytes) 2024-09-16T13:17:44.708 INFO:teuthology.orchestra.run.smithi098.stdout:3229 write 0x2b540a2 thru 0x2b5c586 (0x84e5 bytes) 2024-09-16T13:17:44.711 INFO:teuthology.orchestra.run.smithi098.stdout:3230 clone 19 order 18 su 65536 sc 7 2024-09-16T13:17:44.756 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:44.748+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:44.757 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:44.757 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:44.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:44.857 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:44.851+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:44.861 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:44.861 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:44.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:44.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:44.958 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:44.951+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:44.973 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:17:45.017 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:45.018 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:45.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:45.059 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:45.051+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:45.079 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:45.100 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:45.117 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:45.122 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:45.123 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:45.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:45.136 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:45.158 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:45.160 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:45.152+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:45.179 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:45.200 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:45.223 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:45.261 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:45.252+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:45.277 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:45.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:45.362 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:45.352+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:45.381 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:45.381 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:45.382 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:45.382 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:45.382 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:45.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:45.463 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:45.455+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:45.489 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:45.489 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:45.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:45.489 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:45.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.489 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11487 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 430080~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.996949468, 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014286111000000001, 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11487 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486860+0000", 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.04e-06 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486866+0000", 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0629999999999996e-06 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486881+0000", 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5023e-05 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486889+0000", 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8149999999999995e-06 2024-09-16T13:17:45.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.500906+0000", 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014016416 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501008+0000", 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102243 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501146+0000", 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013751099999999999 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11535 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.996266495, 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014337727999999999, 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.494 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11535 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487543+0000", 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023e-06 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487551+0000", 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0309999999999998e-06 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487566+0000", 2024-09-16T13:17:45.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5000999999999999e-05 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487578+0000", 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1066e-05 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501655+0000", 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014077151 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501747+0000", 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2670999999999994e-05 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501880+0000", 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013278500000000001 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.498 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11541 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 413696~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.996221944999998, 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014490381, 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11541 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487588+0000", 2024-09-16T13:17:45.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487592+0000", 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4920000000000004e-06 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487607+0000", 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4926000000000001e-05 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487616+0000", 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6829999999999997e-06 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501906+0000", 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0142898 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501959+0000", 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2985000000000002e-05 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502077+0000", 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118495 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11542 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 393216~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.996183420000001, 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014676434, 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.503 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11542 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487626+0000", 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.034e-06 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487631+0000", 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7659999999999998e-06 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487646+0000", 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4786e-05 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487655+0000", 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6489999999999997e-06 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502110+0000", 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014455749 2024-09-16T13:17:45.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502209+0000", 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8608000000000002e-05 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502302+0000", 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2842000000000003e-05 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.508 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11544 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 466944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.508 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:45.508 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.996142459000001, 2024-09-16T13:17:45.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014846796000000001, 2024-09-16T13:17:45.508 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.508 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.508 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.508 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11544 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487667+0000", 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.569999999999999e-07 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487672+0000", 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2459999999999997e-06 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487686+0000", 2024-09-16T13:17:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4847e-05 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487695+0000", 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9109999999999999e-06 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502337+0000", 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014641861000000001 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502440+0000", 2024-09-16T13:17:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010282599999999999 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502513+0000", 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3147999999999998e-05 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11547 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.996103332000001, 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015113645, 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11547 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487707+0000", 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.015e-06 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487711+0000", 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8289999999999998e-06 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487727+0000", 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5497000000000001e-05 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487736+0000", 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1409999999999994e-06 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502557+0000", 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014821212 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502675+0000", 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117741 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502819+0000", 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014421 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11549 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 413696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.996062909999999, 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016244525999999999, 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11549 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487747+0000", 2024-09-16T13:17:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.094e-06 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487752+0000", 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.758e-06 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487770+0000", 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8502999999999999e-05 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487779+0000", 2024-09-16T13:17:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1579999999999994e-06 2024-09-16T13:17:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502868+0000", 2024-09-16T13:17:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015088811000000001 2024-09-16T13:17:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502916+0000", 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8109000000000003e-05 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503990+0000", 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010740929999999999 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11578 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 409600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.996012468, 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015410814, 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11578 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487798+0000", 2024-09-16T13:17:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.646e-06 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487804+0000", 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487821+0000", 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7677999999999999e-05 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487830+0000", 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9090000000000006e-06 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503033+0000", 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015202337999999999 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503123+0000", 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0414000000000005e-05 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503207+0000", 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4172000000000007e-05 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11580 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.995968831999999, 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015544980999999999, 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11580 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487841+0000", 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487846+0000", 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3919999999999996e-06 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487861+0000", 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4946000000000001e-05 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487869+0000", 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8750000000000006e-06 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503240+0000", 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015370511999999999 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503334+0000", 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4265000000000006e-05 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503385+0000", 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0773999999999999e-05 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11617 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 430080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.994649133999999, 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376203000000001, 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11617 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489162+0000", 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3089999999999998e-06 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489171+0000", 2024-09-16T13:17:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3430000000000002e-06 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489198+0000", 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6449999999999999e-05 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489216+0000", 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8057999999999999e-05 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503402+0000", 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014186196999999999 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503454+0000", 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1989e-05 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503536+0000", 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1856999999999999e-05 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11624 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 503808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.994446046, 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376676, 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11624 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489365+0000", 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.795e-06 2024-09-16T13:17:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489372+0000", 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4030000000000001e-06 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489392+0000", 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0129e-05 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489404+0000", 2024-09-16T13:17:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1555e-05 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503558+0000", 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014154771999999999 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503644+0000", 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5691000000000003e-05 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503740+0000", 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5330999999999999e-05 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11630 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.994392709, 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014893777, 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11630 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489417+0000", 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6800000000000009e-07 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489435+0000", 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7818000000000001e-05 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489450+0000", 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5072000000000001e-05 2024-09-16T13:17:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489459+0000", 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3009999999999995e-06 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503787+0000", 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01432719 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503889+0000", 2024-09-16T13:17:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102604 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504310+0000", 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042082399999999998 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11632 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.994339596, 2024-09-16T13:17:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01524066, 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11632 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489470+0000", 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.78e-07 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489475+0000", 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9400000000000001e-06 2024-09-16T13:17:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489488+0000", 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3176999999999999e-05 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489497+0000", 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4919999999999993e-06 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504364+0000", 2024-09-16T13:17:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014866846 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504485+0000", 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012142999999999999 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504710+0000", 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000224897 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11639 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.994297825, 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015375941000000001, 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11639 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489512+0000", 2024-09-16T13:17:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489517+0000", 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.938e-06 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489532+0000", 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.47e-05 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489540+0000", 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2949999999999994e-06 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504765+0000", 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015225449 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504832+0000", 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6929999999999998e-05 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504887+0000", 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4681e-05 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11641 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.994251315, 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015522312999999999, 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11641 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489559+0000", 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.066e-06 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489564+0000", 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9370000000000003e-06 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489579+0000", 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5466000000000001e-05 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489588+0000", 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504925+0000", 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015337353 2024-09-16T13:17:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505025+0000", 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9770000000000002e-05 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505080+0000", 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4792e-05 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11667 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 16384~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.993890545999999, 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015353510000000001, 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11667 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489920+0000", 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.84e-06 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489928+0000", 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6969999999999993e-06 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489953+0000", 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5154999999999999e-05 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489965+0000", 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2292e-05 2024-09-16T13:17:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505115+0000", 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015149793999999999 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505184+0000", 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8858999999999999e-05 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505272+0000", 2024-09-16T13:17:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7873000000000004e-05 2024-09-16T13:17:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11672 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 36864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.993830141, 2024-09-16T13:17:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015615053, 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11672 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489980+0000", 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489985+0000", 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490000+0000", 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490010+0000", 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.183e-06 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505335+0000", 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015325167000000001 2024-09-16T13:17:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505471+0000", 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013581699999999999 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505594+0000", 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123168 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11676 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.993789397, 2024-09-16T13:17:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015779395000000002, 2024-09-16T13:17:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11676 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490021+0000", 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.575e-06 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490026+0000", 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5170000000000001e-06 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490041+0000", 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4955e-05 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490049+0000", 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5520000000000005e-06 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505647+0000", 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015597489000000001 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505743+0000", 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6908000000000001e-05 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505799+0000", 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5399000000000003e-05 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11678 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.993750124000002, 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015992643000000001, 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11678 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490060+0000", 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3400000000000001e-06 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490065+0000", 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4830000000000001e-06 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490079+0000", 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4854999999999999e-05 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490088+0000", 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1119999999999996e-06 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505849+0000", 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015761718000000001 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505966+0000", 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011645399999999999 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506051+0000", 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5680999999999995e-05 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11707 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 53248~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.993530995, 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016106869999999999, 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11707 2024-09-16T13:17:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490280+0000", 2024-09-16T13:17:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7659999999999999e-06 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490287+0000", 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.503e-06 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490307+0000", 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9989000000000001e-05 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490318+0000", 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0967e-05 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506171+0000", 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015853367 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506265+0000", 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3190999999999996e-05 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506385+0000", 2024-09-16T13:17:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120087 2024-09-16T13:17:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.580 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:45.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:45.582 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:45.575+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:45.682 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:45.675+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:45.732 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:45.733 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:45.733 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:45.733 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:45.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:45.783 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:45.776+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:45.841 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:45.841 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:45.842 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:45.842 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:45.842 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:45.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:45.884 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:45.875+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:45.950 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:45.950 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:45.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:45.950 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:45.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.950 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:45.950 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.918860843999994, 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:17:45.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:17:45.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.918713437999997, 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.957 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:17:45.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:17:45.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:45.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:17:45.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:17:45.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:45.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:17:45.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:17:45.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.918686094999998, 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:45.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:17:45.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.360010833000004, 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.969 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:17:45.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:45.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.359923241000004, 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:45.975 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:17:45.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.359753319000006, 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.981 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:45.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:17:45.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2259 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3014656~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208424743000002, 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016650893999999999, 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.987 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2259 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735831+0000", 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9530000000000005e-06 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735836+0000", 2024-09-16T13:17:45.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7450000000000003e-06 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735839+0000", 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.915e-06 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752296+0000", 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016457241000000001 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752360+0000", 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4108000000000004e-05 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752473+0000", 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112305 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2260 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:45.991 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208404821999999, 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016799073000000001, 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2260 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735843+0000", 2024-09-16T13:17:45.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3480000000000001e-06 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735847+0000", 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5990000000000002e-06 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735849+0000", 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5859999999999999e-06 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752509+0000", 2024-09-16T13:17:45.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016659377 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752559+0000", 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9774000000000002e-05 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752641+0000", 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2057000000000004e-05 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:45.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2261 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208392633000003, 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016960698, 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2261 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.996 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:45.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.03e-07 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735855+0000", 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735860+0000", 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1110000000000001e-06 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:45.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735862+0000", 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752695+0000", 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01683343 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752744+0000", 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8414000000000001e-05 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:45.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752814+0000", 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0623000000000005e-05 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2262 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1048576~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208382018999998, 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017224107999999998, 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.000 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2262 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735865+0000", 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.91e-07 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735866+0000", 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1480000000000001e-06 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735870+0000", 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735877+0000", 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4370000000000001e-06 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752871+0000", 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016994061000000001 2024-09-16T13:17:46.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752962+0000", 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0470000000000006e-05 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753089+0000", 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126642 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2263 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208366560000002, 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017392088999999999, 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2263 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:46.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-07 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735881+0000", 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735885+0000", 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4180000000000001e-06 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735887+0000", 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-06 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753116+0000", 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017228963 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753184+0000", 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7374999999999998e-05 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753272+0000", 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8323000000000001e-05 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2264 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2064384~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208356709, 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017630541999999999, 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.009 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2264 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735891+0000", 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735895+0000", 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735897+0000", 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4669999999999998e-06 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753316+0000", 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017418685 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753403+0000", 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7590000000000007e-05 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753520+0000", 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116863 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.013 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2265 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1081344~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208346827, 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017739263000000002, 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2265 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735901+0000", 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735904+0000", 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.304e-06 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735907+0000", 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3599999999999999e-06 2024-09-16T13:17:46.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753534+0000", 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017627446000000001 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753576+0000", 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1474999999999998e-05 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753639+0000", 2024-09-16T13:17:46.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3188999999999994e-05 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2266 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2097152~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208337594, 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017970893000000002, 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.018 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2266 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735914+0000", 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1699999999999996e-06 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735918+0000", 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6859999999999999e-06 2024-09-16T13:17:46.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735920+0000", 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4870000000000001e-06 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753685+0000", 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017764582000000001 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753771+0000", 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6259000000000002e-05 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753880+0000", 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010844300000000001 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.022 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2267 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208323692999997, 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018110804000000001, 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2267 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735924+0000", 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735928+0000", 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.416e-06 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735930+0000", 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753898+0000", 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017967852999999999 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753955+0000", 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6662000000000001e-05 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754034+0000", 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8904000000000002e-05 2024-09-16T13:17:46.026 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2268 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3080192~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208314207999997, 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018252866, 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.027 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2268 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735934+0000", 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0950000000000001e-06 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735938+0000", 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735941+0000", 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8750000000000002e-06 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754071+0000", 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018129941 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754120+0000", 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8359000000000002e-05 2024-09-16T13:17:46.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754185+0000", 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5358999999999995e-05 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2269 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208302828999997, 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018533516, 2024-09-16T13:17:46.031 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2269 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735945+0000", 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1149999999999999e-06 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735948+0000", 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5250000000000001e-06 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735951+0000", 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9909999999999998e-06 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754270+0000", 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018318081999999999 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754325+0000", 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5648e-05 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754477+0000", 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015188199999999999 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2270 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1114112~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208292360000002, 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019103282999999999, 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2270 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735955+0000", 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735956+0000", 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.096e-06 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735959+0000", 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5140000000000002e-06 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735963+0000", 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.053e-06 2024-09-16T13:17:46.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754822+0000", 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018858883 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754941+0000", 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118789 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755057+0000", 2024-09-16T13:17:46.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116272 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2271 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1146880~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208270548000002, 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019332309999999998, 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.040 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2271 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3500000000000002e-07 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735982+0000", 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735987+0000", 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1930000000000003e-06 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735993+0000", 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4659999999999996e-06 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755081+0000", 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019088187 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755194+0000", 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011223299999999999 2024-09-16T13:17:46.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755308+0000", 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114663 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2272 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 163840~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.208248718, 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019476291, 2024-09-16T13:17:46.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2272 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736003+0000", 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2279999999999998e-06 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736007+0000", 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.506e-06 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736011+0000", 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9280000000000003e-06 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755336+0000", 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019325189999999999 2024-09-16T13:17:46.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755393+0000", 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7293999999999997e-05 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755474+0000", 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0635000000000003e-05 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.048 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:46.049 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:46.049 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:46.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:46.051 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:46.040+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:46.090 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone18 from 0x4a73da0 (overlap 0x4a73da0) to 0xee3f86 2024-09-16T13:17:46.151 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:46.143+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:46.154 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:46.154 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:46.154 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:46.154 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:46.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:46.216 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #17, image image_client.0-clone17 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent18 2024-09-16T13:17:46.252 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:46.243+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:46.316 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:46.317 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:46.317 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:46.317 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:46.317 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:46.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:46.352 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:46.343+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:46.440 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11500 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 356352~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.938321692999999, 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014537983000000001, 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11500 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496474+0000", 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.801e-06 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496480+0000", 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1140000000000002e-06 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496484+0000", 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8770000000000003e-06 2024-09-16T13:17:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507573+0000", 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011088473 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507624+0000", 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1591999999999997e-05 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511010+0000", 2024-09-16T13:17:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033857380000000001 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11501 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 430080~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.938305325999998, 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014726846, 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11501 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496489+0000", 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3700000000000001e-07 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496491+0000", 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9570000000000001e-06 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496497+0000", 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2550000000000003e-06 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496501+0000", 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8639999999999998e-06 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511060+0000", 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014559462 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511125+0000", 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4399999999999993e-05 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511215+0000", 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0470999999999995e-05 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11530 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 339968~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.937371138, 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014141661, 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11530 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497423+0000", 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-07 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497434+0000", 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1195e-05 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497442+0000", 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.323e-06 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497446+0000", 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1049999999999997e-06 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511275+0000", 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013828872000000001 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511389+0000", 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114493 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511564+0000", 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017508700000000001 2024-09-16T13:17:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11533 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 413696~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.937342053999998, 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014296237999999999, 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11533 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.51e-07 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497458+0000", 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9139999999999996e-06 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497464+0000", 2024-09-16T13:17:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0469999999999999e-06 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497467+0000", 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5030000000000002e-06 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511593+0000", 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014125692 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511669+0000", 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5950000000000003e-05 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511748+0000", 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8681000000000001e-05 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11551 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 335872~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.937286026999999, 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014415314, 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11551 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497510+0000", 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.012e-06 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497516+0000", 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7880000000000003e-06 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497524+0000", 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9049999999999997e-06 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511769+0000", 2024-09-16T13:17:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245809 2024-09-16T13:17:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511818+0000", 2024-09-16T13:17:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8092e-05 2024-09-16T13:17:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511923+0000", 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010532299999999999 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11553 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 413696~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.937263177999998, 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014561233999999999, 2024-09-16T13:17:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11553 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497531+0000", 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1899999999999998e-07 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497533+0000", 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9149999999999999e-06 2024-09-16T13:17:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497539+0000", 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.088e-06 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497550+0000", 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0834e-05 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511962+0000", 2024-09-16T13:17:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411947 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512018+0000", 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6755999999999997e-05 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512092+0000", 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3275000000000006e-05 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11576 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 335872~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499621+0000", 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.935172202, 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013787382000000001, 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11576 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499621+0000", 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499621+0000", 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499622+0000", 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6400000000000003e-07 2024-09-16T13:17:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499624+0000", 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499630+0000", 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9429999999999997e-06 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499634+0000", 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513288+0000", 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013654463 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513342+0000", 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4357000000000002e-05 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513409+0000", 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6456000000000002e-05 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11577 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 409600~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.935155934000001, 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013920979999999999, 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11577 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499640+0000", 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1430000000000001e-06 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499646+0000", 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1029999999999998e-06 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499650+0000", 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6370000000000001e-06 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513442+0000", 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013791805000000001 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513494+0000", 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2074999999999999e-05 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513559+0000", 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4874000000000003e-05 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11628 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 471040~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.934798740000002, 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013902296999999999, 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11628 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499995+0000", 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3599999999999999e-07 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499997+0000", 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.181e-06 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500004+0000", 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4609999999999996e-06 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500008+0000", 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0629999999999999e-06 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513576+0000", 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013567818000000001 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513626+0000", 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0013999999999999e-05 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513897+0000", 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027132399999999998 2024-09-16T13:17:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11637 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.934263503, 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013965546000000001, 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11637 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500531+0000", 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2699999999999999e-07 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500534+0000", 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.055e-06 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500542+0000", 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6389999999999998e-06 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500553+0000", 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0974e-05 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514249+0000", 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013696086 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514308+0000", 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8269000000000001e-05 2024-09-16T13:17:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514496+0000", 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018799600000000001 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11638 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.934234021000002, 2024-09-16T13:17:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014058684, 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11638 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500563+0000", 2024-09-16T13:17:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0790000000000001e-06 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500580+0000", 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6472000000000001e-05 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500586+0000", 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2199999999999997e-06 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514522+0000", 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013935774 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514564+0000", 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2549000000000002e-05 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514618+0000", 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3942e-05 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11642 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.934149187999999, 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014093902, 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11642 2024-09-16T13:17:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500645+0000", 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5200000000000002e-07 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500656+0000", 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1372999999999999e-05 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500664+0000", 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9019999999999999e-06 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500677+0000", 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2954e-05 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514649+0000", 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013971417 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514692+0000", 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3154000000000001e-05 2024-09-16T13:17:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514738+0000", 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6650000000000002e-05 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11646 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.934110925999999, 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245793, 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11646 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500685+0000", 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2019999999999998e-06 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500698+0000", 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.279e-05 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500702+0000", 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7629999999999998e-06 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514772+0000", 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014070384999999999 2024-09-16T13:17:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514865+0000", 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2980999999999996e-05 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514928+0000", 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3114000000000006e-05 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11648 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.934087796, 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411604999999999, 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11648 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9900000000000001e-07 2024-09-16T13:17:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500713+0000", 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.793e-06 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500719+0000", 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500722+0000", 2024-09-16T13:17:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3809999999999999e-06 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514973+0000", 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014250924 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515031+0000", 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7864e-05 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515117+0000", 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6697e-05 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11654 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.933720961999999, 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014438998, 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11654 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8699999999999995e-07 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501075+0000", 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501082+0000", 2024-09-16T13:17:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.844e-06 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501086+0000", 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7749999999999999e-06 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515152+0000", 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014065891000000001 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515232+0000", 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0461000000000001e-05 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515512+0000", 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027934599999999998 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11675 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.933258368000001, 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014239586, 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11675 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501536+0000", 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501539+0000", 2024-09-16T13:17:46.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3529999999999999e-06 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501546+0000", 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.109e-06 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501551+0000", 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5109999999999997e-06 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515586+0000", 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014035334999999999 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515689+0000", 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010244200000000001 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515775+0000", 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6150999999999996e-05 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.511 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11679 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.933208444000002, 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014462922, 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11679 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501586+0000", 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501588+0000", 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3760000000000002e-06 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501594+0000", 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9950000000000002e-06 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501598+0000", 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7239999999999998e-06 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515840+0000", 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014242546999999999 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515949+0000", 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108591 2024-09-16T13:17:46.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516048+0000", 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9123000000000006e-05 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11686 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.933191871999998, 2024-09-16T13:17:46.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014648357000000001, 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11686 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501604+0000", 2024-09-16T13:17:46.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.846e-06 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501609+0000", 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3630000000000002e-06 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501613+0000", 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3749999999999999e-06 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516077+0000", 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014464351 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516173+0000", 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5742000000000006e-05 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516250+0000", 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7312999999999997e-05 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11689 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.933177308000001, 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014770922000000001, 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11689 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.521 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501617+0000", 2024-09-16T13:17:46.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501618+0000", 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6780000000000001e-06 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501624+0000", 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3750000000000002e-06 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501627+0000", 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5149999999999998e-06 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516279+0000", 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014651928999999999 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516336+0000", 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6724000000000001e-05 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516387+0000", 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1334e-05 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11778 3.1 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.91584318, 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022495198000000001, 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11778 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518952+0000", 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.711e-06 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522261+0000", 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033087429999999998 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522296+0000", 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5335e-05 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522306+0000", 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0074999999999999e-05 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522379+0000", 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2554999999999999e-05 2024-09-16T13:17:46.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522572+0000", 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019323100000000001 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.523266+0000", 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069379000000000001 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.534086+0000", 2024-09-16T13:17:46.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010820112999999999 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541312+0000", 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072264440000000003 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541394+0000", 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1710000000000002e-05 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541446+0000", 2024-09-16T13:17:46.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1490999999999999e-05 2024-09-16T13:17:46.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.531 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.531 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.531 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:46.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:46.533 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:46.524+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:46.616 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:46.617 INFO:teuthology.orchestra.run.smithi098.stdout:3232 read 0x2451983 thru 0x24535e4 (0x1c62 bytes) 2024-09-16T13:17:46.618 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:46.618 INFO:teuthology.orchestra.run.smithi098.stdout:3234 read 0x1c55f43 thru 0x1c5a39c (0x445a bytes) 2024-09-16T13:17:46.620 INFO:teuthology.orchestra.run.smithi098.stdout:3235 punch from 0x2a32cf3 to 0x2a3394e, (0xc5b bytes) 2024-09-16T13:17:46.624 INFO:teuthology.orchestra.run.smithi098.stdout:3238 write 0x249c86d thru 0x24a3864 (0x6ff8 bytes) 2024-09-16T13:17:46.633 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:46.624+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:46.633 INFO:teuthology.orchestra.run.smithi098.stdout:3239 read 0x48dcf71 thru 0x48eb24d (0xe2dd bytes) 2024-09-16T13:17:46.634 INFO:teuthology.orchestra.run.smithi098.stdout:3242 read 0x10365e thru 0x10d47a (0x9e1d bytes) 2024-09-16T13:17:46.645 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:46.645 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:46.645 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:46.645 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:46.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:46.659 INFO:teuthology.orchestra.run.smithi098.stdout:3243 writesame 0xda1b2a0 thru 0xda232a9 data_size 0x1b(0x800a bytes) 2024-09-16T13:17:46.659 INFO:teuthology.orchestra.run.smithi098.stdout:3245 writesame 0x2f263db thru 0x2f26d93 data_size 0x1(0x9b9 bytes) 2024-09-16T13:17:46.668 INFO:teuthology.orchestra.run.smithi098.stdout:3246 write 0xd84db30 thru 0xd85d03f (0xf510 bytes) 2024-09-16T13:17:46.677 INFO:teuthology.orchestra.run.smithi098.stdout:3247 read 0x791cc5c thru 0x7928331 (0xb6d6 bytes) 2024-09-16T13:17:46.677 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:46.677 INFO:teuthology.orchestra.run.smithi098.stdout:3250 read 0x84db31f thru 0x84de4b7 (0x3199 bytes) 2024-09-16T13:17:46.678 INFO:teuthology.orchestra.run.smithi098.stdout:3251 punch from 0xcc683bf to 0xcc6fbe2, (0x7823 bytes) 2024-09-16T13:17:46.678 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:46.678 INFO:teuthology.orchestra.run.smithi098.stdout:3254 writesame 0xa447d46 thru 0xa456ede data_size 0x1(0xf199 bytes) 2024-09-16T13:17:46.734 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:46.727+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:46.737 INFO:teuthology.orchestra.run.smithi098.stdout:3255 punch from 0x5cc3e45 to 0x5cd3467, (0xf622 bytes) 2024-09-16T13:17:46.737 INFO:teuthology.orchestra.run.smithi098.stdout:3256 punch from 0xa8b5032 to 0xa8b6460, (0x142e bytes) 2024-09-16T13:17:46.737 INFO:teuthology.orchestra.run.smithi098.stdout:3257 write 0x72183d5 thru 0x721e0b7 (0x5ce3 bytes) 2024-09-16T13:17:46.744 INFO:teuthology.orchestra.run.smithi098.stdout:3258 write 0x1029aab thru 0x1038f28 (0xf47e bytes) 2024-09-16T13:17:46.753 INFO:teuthology.orchestra.run.smithi098.stdout:3259 write 0x7d63fc9 thru 0x7d70172 (0xc1aa bytes) 2024-09-16T13:17:46.761 INFO:teuthology.orchestra.run.smithi098.stdout:3260 punch from 0xb13a2c9 to 0xb1485ce, (0xe305 bytes) 2024-09-16T13:17:46.769 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:46.770 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:46.770 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:46.770 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:46.770 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:46.771 INFO:teuthology.orchestra.run.smithi098.stdout:3263 write 0xe45c954 thru 0xe45e3f1 (0x1a9e bytes) 2024-09-16T13:17:46.777 INFO:teuthology.orchestra.run.smithi098.stdout:3264 write 0x6710d79 thru 0x67159b4 (0x4c3c bytes) 2024-09-16T13:17:46.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:46.783 INFO:teuthology.orchestra.run.smithi098.stdout:3265 write 0x26c2e6 thru 0x276a27 (0xa742 bytes) 2024-09-16T13:17:46.809 INFO:teuthology.orchestra.run.smithi098.stdout:3266 trunc from 0xe45e3f2 to 0x45ad8a4 2024-09-16T13:17:46.835 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:46.828+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:46.848 INFO:teuthology.orchestra.run.smithi098.stdout:3267 read 0x10b3fca thru 0x10b764b (0x3682 bytes) 2024-09-16T13:17:46.849 INFO:teuthology.orchestra.run.smithi098.stdout:3268 read 0xc55627 thru 0xc5c339 (0x6d13 bytes) 2024-09-16T13:17:46.849 INFO:teuthology.orchestra.run.smithi098.stdout:3270 read 0x26d828e thru 0x26e66bb (0xe42e bytes) 2024-09-16T13:17:46.850 INFO:teuthology.orchestra.run.smithi098.stdout:3272 writesame 0x4b7c85 thru 0x4c097d data_size 0x1(0x8cf9 bytes) 2024-09-16T13:17:46.882 INFO:teuthology.orchestra.run.smithi098.stdout:3273 writesame 0x1a5ebce thru 0x1a6333f data_size 0x3e(0x4772 bytes) 2024-09-16T13:17:46.885 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6966 3.0 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.326529600000001, 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095509229999999994, 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6966 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554532+0000", 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2000000000000001e-07 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554534+0000", 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9850000000000001e-06 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554537+0000", 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.664e-06 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554541+0000", 2024-09-16T13:17:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2909999999999999e-06 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557170+0000", 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026295310000000001 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557345+0000", 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017473200000000001 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557618+0000", 2024-09-16T13:17:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027282800000000002 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563504+0000", 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058862649999999999 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563923+0000", 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000419004 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564003+0000", 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0072000000000002e-05 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564082+0000", 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9231000000000001e-05 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11223 3.8 e40/23 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head v 40'368, mlcod=39'365)", 2024-09-16T13:17:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.414358516, 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01312405, 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466704+0000", 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.792e-06 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471144+0000", 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044394830000000001 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471180+0000", 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5884999999999999e-05 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471194+0000", 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4258e-05 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471273+0000", 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9159999999999994e-05 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471305+0000", 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2236000000000001e-05 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479753+0000", 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0084474049999999998 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479826+0000", 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3831000000000005e-05 2024-09-16T13:17:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11776 3.1a e40/23 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head v 40'72, mlcod=40'70)", 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.353461718999998, 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595333, 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527601+0000", 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.522e-06 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536775+0000", 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091746569999999993 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536818+0000", 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2435000000000003e-05 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536828+0000", 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0624e-05 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536931+0000", 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010280000000000001 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536961+0000", 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0060999999999999e-05 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544146+0000", 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071846339999999996 2024-09-16T13:17:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544195+0000", 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8600000000000002e-05 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13719 3.19 3:99fbe691:::rbd_data.112df1cc59bd.000000000000000c:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.600292148000001, 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011713708999999999, 2024-09-16T13:17:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13719 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280771+0000", 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9240000000000001e-06 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280782+0000", 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1642e-05 2024-09-16T13:17:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280806+0000", 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4119000000000001e-05 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280815+0000", 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7150000000000004e-06 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:17:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012396900000000001 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281142+0000", 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000203298 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281540+0000", 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039763500000000002 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.286598+0000", 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050579090000000002 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292380+0000", 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057823390000000001 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292433+0000", 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2612999999999999e-05 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292482+0000", 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9546000000000003e-05 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13720 3.19 3:99acede8:::rbd_data.112df1cc59bd.000000000000000d:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.600228955999999, 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011830879000000001, 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13720 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280836+0000", 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.703e-06 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280844+0000", 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280849+0000", 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.254e-06 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282739+0000", 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001889672 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282875+0000", 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013641 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.283217+0000", 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00034147099999999998 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.288528+0000", 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053110140000000002 2024-09-16T13:17:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292558+0000", 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040304429999999999 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292620+0000", 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1588999999999996e-05 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292663+0000", 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2922e-05 2024-09-16T13:17:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13726 3.19 3:9e45447e:::rbd_data.112df1cc59bd.0000000000000034:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.600128243, 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012171434, 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13726 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280934+0000", 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9299999999999996e-07 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280950+0000", 2024-09-16T13:17:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1141000000000001e-05 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280955+0000", 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5560000000000001e-06 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284230+0000", 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032752789999999999 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284286+0000", 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6456999999999998e-05 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284430+0000", 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014382899999999999 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.291179+0000", 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067490099999999997 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293014+0000", 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018347719999999999 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293069+0000", 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5325999999999998e-05 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293104+0000", 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4892999999999999e-05 2024-09-16T13:17:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14647 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 3801088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245930305999998, 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0094151970000000001, 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14647 2024-09-16T13:17:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635132+0000", 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4300000000000001e-07 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635136+0000", 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3660000000000002e-06 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635150+0000", 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4491999999999999e-05 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635159+0000", 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2630000000000003e-06 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644405+0000", 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0092462220000000001 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644476+0000", 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0958999999999996e-05 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644546+0000", 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9951999999999997e-05 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14648 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4325376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245892618999999, 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096264239999999997, 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14648 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635169+0000", 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.071e-06 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635173+0000", 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635188+0000", 2024-09-16T13:17:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4389000000000001e-05 2024-09-16T13:17:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635198+0000", 2024-09-16T13:17:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0566e-05 2024-09-16T13:17:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644579+0000", 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093805399999999997 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644670+0000", 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1240000000000001e-05 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644795+0000", 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012454 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14662 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1703936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245647052999999, 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096453119999999996, 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14662 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635415+0000", 2024-09-16T13:17:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5e-06 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635421+0000", 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730000000000001e-06 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635436+0000", 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4909000000000001e-05 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635450+0000", 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3183999999999999e-05 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644833+0000", 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093835959999999993 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644914+0000", 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0624000000000006e-05 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645059+0000", 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014542600000000001 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14663 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2228224~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245601891, 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098295339999999991, 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14663 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635460+0000", 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1500000000000004e-07 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635463+0000", 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.478e-06 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635472+0000", 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4230000000000005e-06 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635498+0000", 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6543e-05 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645115+0000", 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096171710000000007 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645193+0000", 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7245999999999999e-05 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645289+0000", 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5958000000000005e-05 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14664 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245555401000001, 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010033297, 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14664 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635506+0000", 2024-09-16T13:17:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5099999999999998e-07 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635511+0000", 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7820000000000001e-06 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635520+0000", 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5969999999999992e-06 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635529+0000", 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3249999999999997e-06 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645337+0000", 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098082589999999997 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645427+0000", 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9922999999999999e-05 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645539+0000", 2024-09-16T13:17:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011156 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14665 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245526172999998, 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010434005999999999, 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14665 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635538+0000", 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8329999999999998e-06 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635546+0000", 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0760000000000003e-06 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635551+0000", 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6160000000000004e-06 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645677+0000", 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010126546 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645821+0000", 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014406799999999999 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645969+0000", 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014730900000000001 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14678 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 0~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245266729000001, 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012023611999999999, 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14678 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635797+0000", 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7039999999999999e-06 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635810+0000", 2024-09-16T13:17:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3343000000000001e-05 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635845+0000", 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4751e-05 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635863+0000", 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8210999999999999e-05 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646005+0000", 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010141766999999999 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646070+0000", 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5324e-05 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647818+0000", 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017475119999999999 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14679 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1835008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245179022999999, 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012069985, 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14679 2024-09-16T13:17:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635883+0000", 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3239999999999999e-06 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635889+0000", 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635905+0000", 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6022999999999999e-05 2024-09-16T13:17:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635911+0000", 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3740000000000003e-06 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647849+0000", 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011937906999999999 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647895+0000", 2024-09-16T13:17:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5507999999999999e-05 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647952+0000", 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7132000000000001e-05 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14680 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2359296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245142740999999, 2024-09-16T13:17:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012172090999999999, 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14680 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635919+0000", 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8299999999999995e-07 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635922+0000", 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.027e-06 2024-09-16T13:17:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635931+0000", 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6230000000000003e-06 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635936+0000", 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0069999999999997e-06 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647981+0000", 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012045273 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648033+0000", 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648090+0000", 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7154000000000002e-05 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14681 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4718592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245119744, 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012312528, 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14681 2024-09-16T13:17:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635942+0000", 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7599999999999996e-07 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635959+0000", 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6954999999999999e-05 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635968+0000", 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8929999999999994e-06 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635973+0000", 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0170000000000004e-06 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648120+0000", 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012146847000000001 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648170+0000", 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0368000000000003e-05 2024-09-16T13:17:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648254+0000", 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3671999999999994e-05 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14694 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245082669999999, 2024-09-16T13:17:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012792843999999999, 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14694 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635979+0000", 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0699999999999996e-07 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635982+0000", 2024-09-16T13:17:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8200000000000001e-06 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635990+0000", 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0239999999999997e-06 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635995+0000", 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648296+0000", 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0123009 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648397+0000", 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101042 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648771+0000", 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037435099999999998 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14695 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1966080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245060651999999, 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012992136, 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14695 2024-09-16T13:17:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636001+0000", 2024-09-16T13:17:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0999999999999998e-07 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636004+0000", 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5959999999999998e-06 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636012+0000", 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636016+0000", 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9269999999999996e-06 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648804+0000", 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012787543 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648873+0000", 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9263000000000004e-05 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648992+0000", 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119089 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14696 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2490368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245038960999999, 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013132612, 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14696 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636023+0000", 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7300000000000005e-07 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636025+0000", 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.407e-06 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636033+0000", 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0129999999999993e-06 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636042+0000", 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1540000000000008e-06 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649027+0000", 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012985096 2024-09-16T13:17:46.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649089+0000", 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1641000000000002e-05 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649155+0000", 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5528e-05 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14697 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.245013060000002, 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013321177999999999, 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14697 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:46.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636049+0000", 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636051+0000", 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.57e-06 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636059+0000", 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9440000000000005e-06 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636067+0000", 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0460000000000006e-06 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649182+0000", 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01311522 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649245+0000", 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2930999999999998e-05 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649369+0000", 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123681 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.979 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.980 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.980 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:46.980 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:46.980 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:46.981 INFO:teuthology.orchestra.run.smithi098.stdout:3274 write 0x2db91c1 thru 0x2dc314b (0x9f8b bytes) 2024-09-16T13:17:46.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:46.982 INFO:teuthology.orchestra.run.smithi098.stdout:3275 write 0xcfe712f thru 0xcff6efc (0xfdce bytes) 2024-09-16T13:17:46.982 INFO:teuthology.orchestra.run.smithi098.stdout:3276 write 0xd8cf976 thru 0xd8dede5 (0xf470 bytes) 2024-09-16T13:17:46.982 INFO:teuthology.orchestra.run.smithi098.stdout:3277 read 0xc8a7d6 thru 0xc92f1b (0x8746 bytes) 2024-09-16T13:17:46.982 INFO:teuthology.orchestra.run.smithi098.stdout:3278 read 0x6c8c8d8 thru 0x6c9b9aa (0xf0d3 bytes) 2024-09-16T13:17:46.983 INFO:teuthology.orchestra.run.smithi098.stdout:3280 writesame 0x5565b00 thru 0x556cc86 data_size 0x1(0x7187 bytes) 2024-09-16T13:17:46.983 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:46.972+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:46.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:17:46.989 INFO:teuthology.orchestra.run.smithi098.stdout:3282 writesame 0x84ec0b3 thru 0x84fb1db data_size 0xd(0xf129 bytes) 2024-09-16T13:17:47.002 INFO:teuthology.orchestra.run.smithi098.stdout:3283 punch from 0x23e5520 to 0x23f211e, (0xcbfe bytes) 2024-09-16T13:17:47.003 INFO:teuthology.orchestra.run.smithi098.stdout:3284 write 0xace3772 thru 0xace5310 (0x1b9f bytes) 2024-09-16T13:17:47.025 INFO:teuthology.orchestra.run.smithi098.stdout:3285 trunc from 0xd8dede6 to 0x6800de5 2024-09-16T13:17:47.050 INFO:teuthology.orchestra.run.smithi098.stdout:3286 read 0x2e8f2c4 thru 0x2e92a6b (0x37a8 bytes) 2024-09-16T13:17:47.051 INFO:teuthology.orchestra.run.smithi098.stdout:3288 punch from 0x6f2b1b to 0x6f589f, (0x2d84 bytes) 2024-09-16T13:17:47.070 INFO:teuthology.orchestra.run.smithi098.stdout:3290 write 0xc01bee5 thru 0xc02494c (0x8a68 bytes) 2024-09-16T13:17:47.082 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:47.075+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:47.089 INFO:teuthology.orchestra.run.smithi098.stdout:3291 write 0xdb3458b thru 0xdb35c28 (0x169e bytes) 2024-09-16T13:17:47.094 INFO:teuthology.orchestra.run.smithi098.stdout:3294 write 0xd8eaee4 thru 0xd8ed394 (0x24b1 bytes) 2024-09-16T13:17:47.104 INFO:teuthology.orchestra.run.smithi098.stdout:3295 trunc from 0xdb35c29 to 0xc53d281 2024-09-16T13:17:47.108 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:47.108 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:47.108 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:47.108 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:47.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:47.129 INFO:teuthology.orchestra.run.smithi098.stdout:3296 write 0xcf77bde thru 0xcf84719 (0xcb3c bytes) 2024-09-16T13:17:47.153 INFO:teuthology.orchestra.run.smithi098.stdout:3298 trunc from 0xcf8471a to 0x186c73c 2024-09-16T13:17:47.183 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:47.175+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:47.207 INFO:teuthology.orchestra.run.smithi098.stdout:3299 write 0xc440de2 thru 0xc44963e (0x885d bytes) 2024-09-16T13:17:47.211 INFO:teuthology.orchestra.run.smithi098.stdout:3302 punch from 0x4ee4c85 to 0x4ee7a63, (0x2dde bytes) 2024-09-16T13:17:47.214 INFO:teuthology.orchestra.run.smithi098.stdout:3303 trunc from 0xc44963f to 0xdcb6882 2024-09-16T13:17:47.222 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:47.222 INFO:teuthology.orchestra.run.smithi098.stdout:3305 read 0x4fb9f88 thru 0x4fbc07f (0x20f8 bytes) 2024-09-16T13:17:47.231 INFO:teuthology.orchestra.run.smithi098.stdout:3306 writesame 0xea46d28 thru 0xea4ed4c data_size 0x51(0x8025 bytes) 2024-09-16T13:17:47.231 INFO:teuthology.orchestra.run.smithi098.stdout:3308 read 0xcafcfed thru 0xcb00c50 (0x3c64 bytes) 2024-09-16T13:17:47.231 INFO:teuthology.orchestra.run.smithi098.stdout:3309 punch from 0x204eb17 to 0x2055faf, (0x7498 bytes) 2024-09-16T13:17:47.231 INFO:teuthology.orchestra.run.smithi098.stdout:3310 read 0x462d34f thru 0x46354d7 (0x8189 bytes) 2024-09-16T13:17:47.231 INFO:teuthology.orchestra.run.smithi098.stdout:3311 read 0x2046f21 thru 0x204d0f3 (0x61d3 bytes) 2024-09-16T13:17:47.231 INFO:teuthology.orchestra.run.smithi098.stdout:3313 read 0x72ea8c6 thru 0x72f89f6 (0xe131 bytes) 2024-09-16T13:17:47.234 INFO:teuthology.orchestra.run.smithi098.stdout:3315 trunc from 0xea4ed4d to 0xcc6c73c 2024-09-16T13:17:47.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:47.244 INFO:teuthology.orchestra.run.smithi098.stdout:3316 write 0x267a33c thru 0x2686700 (0xc3c5 bytes) 2024-09-16T13:17:47.257 INFO:teuthology.orchestra.run.smithi098.stdout:3319 trunc from 0xcc6c73c to 0x975f192 2024-09-16T13:17:47.258 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:47.259 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:47.259 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:47.259 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:47.259 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:47.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:47.273 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:47.273 INFO:teuthology.orchestra.run.smithi098.stdout:3321 write 0x3469c77 thru 0x3472293 (0x861d bytes) 2024-09-16T13:17:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:3322 punch from 0x68317f2 to 0x683a816, (0x9024 bytes) 2024-09-16T13:17:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:3323 read 0x3d78a9a thru 0x3d86f8a (0xe4f1 bytes) 2024-09-16T13:17:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:3325 read 0x680e22b thru 0x680ed17 (0xaed bytes) 2024-09-16T13:17:47.284 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:47.275+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:47.299 INFO:teuthology.orchestra.run.smithi098.stdout:3326 write 0xeb8c6d6 thru 0xeb8f299 (0x2bc4 bytes) 2024-09-16T13:17:47.306 INFO:teuthology.orchestra.run.smithi098.stdout:3327 writesame 0xe8d726c thru 0xe8d74dc data_size 0x1(0x271 bytes) 2024-09-16T13:17:47.312 INFO:teuthology.orchestra.run.smithi098.stdout:3328 write 0xe65339a thru 0xe6600ae (0xcd15 bytes) 2024-09-16T13:17:47.331 INFO:teuthology.orchestra.run.smithi098.stdout:3329 write 0xee490a6 thru 0xee58952 (0xf8ad bytes) 2024-09-16T13:17:47.339 INFO:teuthology.orchestra.run.smithi098.stdout:3331 punch from 0xe15077 to 0xe19b67, (0x4af0 bytes) 2024-09-16T13:17:47.362 INFO:teuthology.orchestra.run.smithi098.stdout:3333 trunc from 0xee58953 to 0x8d4e27 2024-09-16T13:17:47.384 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:47.376+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:47.386 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:47.387 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:1264 3.4 e27/23 3:2779ea43:::rbd_data.112de98401a1.0000000000000009:head v 27'4, mlcod=26'2)", 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 47.882152075, 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026410209999999999, 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501249+0000", 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501250+0000", 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2410000000000001e-06 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501259+0000", 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501277+0000", 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8683999999999999e-05 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501288+0000", 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.065e-05 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501562+0000", 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027421200000000002 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.501618+0000", 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6045000000000003e-05 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503841+0000", 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022230829999999998 2024-09-16T13:17:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:16:59.503890+0000", 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8526000000000003e-05 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.376008386999999, 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:17:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:17:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:17:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.375222888000003, 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:17:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.374055966, 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:47.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:17:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.371181335000003, 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.368167595000003, 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:17:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:17:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.342963392000001, 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:17:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.420 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.576814503000001, 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:17:47.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:17:47.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5115 3.4 e34/23 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head v 34'14, mlcod=31'12)", 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 39.591766397000001, 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029400680000000001, 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791635+0000", 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4200000000000005e-07 2024-09-16T13:17:47.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791643+0000", 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.157e-06 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791660+0000", 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6246e-05 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791666+0000", 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791743+0000", 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7000000000000001e-05 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791756+0000", 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2729e-05 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794523+0000", 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027675949999999999 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794575+0000", 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1082000000000003e-05 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 35.3247122, 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:17:47.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:17:47.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:47.434 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 35.324102760000002, 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:17:47.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6963 3.4 e37/23 3:2713f365:::rbd_data.112d6fb8d387.0000000000000133:head v 37'31, mlcod=37'29)", 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.827538609999998, 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027598190000000002, 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555864+0000", 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:17:47.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555882+0000", 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7816e-05 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555909+0000", 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7838999999999999e-05 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555923+0000", 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3577000000000001e-05 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556083+0000", 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015987299999999999 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556132+0000", 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8655000000000001e-05 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558539+0000", 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024076969999999999 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558622+0000", 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2777999999999999e-05 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.443 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.826818946000003, 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:17:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:17:47.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.825718129999998, 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:17:47.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:17:47.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.452 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.825578489999998, 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:17:47.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:17:47.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:47.457 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 34.825077960000002, 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:17:47.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.461 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.829908574000001, 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:17:47.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:17:47.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:17:47.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:17:47.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:17:47.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:17:47.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:17:47.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.573494733, 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:17:47.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.960931626000001, 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:17:47.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:22854 3.9 e57/23 3:924a2914:::rbd_data.112db5030c67.0000000000000008:head v 57'198, mlcod=57'196)", 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.842411062, 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029464980000000001, 2024-09-16T13:17:47.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540995+0000", 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541018+0000", 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2382999999999999e-05 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541079+0000", 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1773999999999995e-05 2024-09-16T13:17:47.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541094+0000", 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4414e-05 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.542159+0000", 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010653749999999999 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.542208+0000", 2024-09-16T13:17:47.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8517000000000003e-05 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.543797+0000", 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015893210000000001 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.543936+0000", 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000139367 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:47.481 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:47.481 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:47.481 INFO:teuthology.orchestra.run.smithi098.stdout:3335 read 0x5d7ba1 thru 0x5e2d48 (0xb1a8 bytes) 2024-09-16T13:17:47.481 INFO:teuthology.orchestra.run.smithi098.stdout:3339 read 0x44ee52 thru 0x45e0a4 (0xf253 bytes) 2024-09-16T13:17:47.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:47.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:47.483 INFO:teuthology.orchestra.run.smithi098.stdout:3340 write 0xe5267e9 thru 0xe53052b (0x9d43 bytes) 2024-09-16T13:17:47.483 INFO:teuthology.orchestra.run.smithi098.stdout:3341 punch from 0xddb3381 to 0xddb65e6, (0x3265 bytes) 2024-09-16T13:17:47.483 INFO:teuthology.orchestra.run.smithi098.stdout:3343 writesame 0xb191db4 thru 0xb19486d data_size 0x3(0x2aba bytes) 2024-09-16T13:17:47.484 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:47.484 INFO:teuthology.orchestra.run.smithi098.stdout:3344 trunc from 0xe53052c to 0x6bda0ec 2024-09-16T13:17:47.484 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:47.484 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:47.485 INFO:teuthology.orchestra.run.smithi098.stdout:3345 write 0xd84339c thru 0xd84a850 (0x74b5 bytes) 2024-09-16T13:17:47.486 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:47.476+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:47.491 INFO:teuthology.orchestra.run.smithi098.stdout:3346 writesame 0x652f6eb thru 0x6538c7b data_size 0x1(0x9591 bytes) 2024-09-16T13:17:47.498 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:47.520 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:47.529 INFO:teuthology.orchestra.run.smithi098.stdout:3347 read 0x1d47da8 thru 0x1d54c2f (0xce88 bytes) 2024-09-16T13:17:47.530 INFO:teuthology.orchestra.run.smithi098.stdout:3348 read 0x3845554 thru 0x384cf32 (0x79df bytes) 2024-09-16T13:17:47.530 INFO:teuthology.orchestra.run.smithi098.stdout:3349 clone 20 order 19 su 262144 sc 15 2024-09-16T13:17:47.542 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:47.545 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:47.545 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:47.545 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:47.545 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:47.545 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:47.545 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:47.545 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:47.545 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:47.585 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:47.585 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:47.587 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:47.580+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:47.587 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:47.588 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:47.588 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-16T13:17:47.588 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:17:47.588 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:17:47.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-16T13:17:47.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:47.688 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:47.688 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "" 2024-09-16T13:17:47.688 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:47.688 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:47.679+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:47.733 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:47.734 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:47.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:47.789 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:47.779+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:47.837 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:47.838 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:47.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:47.893 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:47.887+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:47.956 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:47.956 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:47.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:47.990 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:47.983+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:17:48.070 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:48.071 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:48.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:48.122 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone19 from 0xd84a851 (overlap 0x8d4e27) to 0x46a713 2024-09-16T13:17:48.129 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:48.123+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:48.181 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:48.182 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:48.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:48.192 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:48.184+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:48.291 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #18, image image_client.0-clone18 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent19 2024-09-16T13:17:48.292 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:48.283+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:48.293 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:48.293 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:48.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:48.326 INFO:teuthology.orchestra.run.smithi098.stdout:3350 read 0x602646 thru 0x60e2e4 (0xbc9f bytes) 2024-09-16T13:17:48.343 INFO:teuthology.orchestra.run.smithi098.stdout:3352 write 0xe7027e5 thru 0xe702eda (0x6f6 bytes) 2024-09-16T13:17:48.348 INFO:teuthology.orchestra.run.smithi098.stdout:3353 write 0x9483359 thru 0x94834d2 (0x17a bytes) 2024-09-16T13:17:48.354 INFO:teuthology.orchestra.run.smithi098.stdout:3354 read 0x25b8119 thru 0x25bb899 (0x3781 bytes) 2024-09-16T13:17:48.354 INFO:teuthology.orchestra.run.smithi098.stdout:3356 write 0xd06ac88 thru 0xd06e102 (0x347b bytes) 2024-09-16T13:17:48.377 INFO:teuthology.orchestra.run.smithi098.stdout:3357 trunc from 0xe702edb to 0x59a2ccc 2024-09-16T13:17:48.393 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:48.384+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:48.403 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:48.403 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:48.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:48.418 INFO:teuthology.orchestra.run.smithi098.stdout:3359 write 0x8ae1a3b thru 0x8aee83e (0xce04 bytes) 2024-09-16T13:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout:3360 punch from 0x26bead9 to 0x26c2e7c, (0x43a3 bytes) 2024-09-16T13:17:48.424 INFO:teuthology.orchestra.run.smithi098.stdout:3361 write 0x40bf193 thru 0x40c8aac (0x991a bytes) 2024-09-16T13:17:48.448 INFO:teuthology.orchestra.run.smithi098.stdout:3362 trunc from 0x8aee83f to 0x174916d 2024-09-16T13:17:48.477 INFO:teuthology.orchestra.run.smithi098.stdout:3364 punch from 0xff53e7 to 0x100284a, (0xd463 bytes) 2024-09-16T13:17:48.494 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:48.484+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:48.507 INFO:teuthology.orchestra.run.smithi098.stdout:3366 write 0xd16bf25 thru 0xd1741bd (0x8299 bytes) 2024-09-16T13:17:48.512 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:48.512 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:48.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:48.536 INFO:teuthology.orchestra.run.smithi098.stdout:3368 trunc from 0xd1741be to 0x267ee41 2024-09-16T13:17:48.581 INFO:teuthology.orchestra.run.smithi098.stdout:3369 write 0xb370166 thru 0xb37f49f (0xf33a bytes) 2024-09-16T13:17:48.595 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:48.584+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:48.600 INFO:teuthology.orchestra.run.smithi098.stdout:3370 trunc from 0xb37f4a0 to 0x37d602a 2024-09-16T13:17:48.615 INFO:teuthology.orchestra.run.smithi098.stdout:3372 writesame 0x303fd01 thru 0x304642f data_size 0xf(0x672f bytes) 2024-09-16T13:17:48.623 INFO:teuthology.orchestra.run.smithi098.stdout:3373 write 0x1fba5fe thru 0x1fbe1a0 (0x3ba3 bytes) 2024-09-16T13:17:48.624 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:48.625 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:48.630 INFO:teuthology.orchestra.run.smithi098.stdout:3374 read 0x207af86 thru 0x207f234 (0x42af bytes) 2024-09-16T13:17:48.630 INFO:teuthology.orchestra.run.smithi098.stdout:3375 read 0x2e3f35b thru 0x2e460f9 (0x6d9f bytes) 2024-09-16T13:17:48.630 INFO:teuthology.orchestra.run.smithi098.stdout:3376 punch from 0x15752c to 0x1626ed, (0xb1c1 bytes) 2024-09-16T13:17:48.630 INFO:teuthology.orchestra.run.smithi098.stdout:3377 read 0x2942e77 thru 0x295021b (0xd3a5 bytes) 2024-09-16T13:17:48.630 INFO:teuthology.orchestra.run.smithi098.stdout:3379 trunc from 0x37d602a to 0x3d3ddcc 2024-09-16T13:17:48.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:48.667 INFO:teuthology.orchestra.run.smithi098.stdout:3380 write 0xaaafe2a thru 0xaab7aab (0x7c82 bytes) 2024-09-16T13:17:48.674 INFO:teuthology.orchestra.run.smithi098.stdout:3381 writesame 0x90ec3 thru 0x9fee3 data_size 0x1f(0xf021 bytes) 2024-09-16T13:17:48.688 INFO:teuthology.orchestra.run.smithi098.stdout:3382 trunc from 0xaab7aac to 0xbe5c467 2024-09-16T13:17:48.696 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:48.687+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:48.698 INFO:teuthology.orchestra.run.smithi098.stdout:3383 write 0x3be363 thru 0x3ca387 (0xc025 bytes) 2024-09-16T13:17:48.710 INFO:teuthology.orchestra.run.smithi098.stdout:3384 read 0x80be6e4 thru 0x80c728c (0x8ba9 bytes) 2024-09-16T13:17:48.711 INFO:teuthology.orchestra.run.smithi098.stdout:3388 write 0x4070f40 thru 0x4077ada (0x6b9b bytes) 2024-09-16T13:17:48.724 INFO:teuthology.orchestra.run.smithi098.stdout:3389 trunc from 0xbe5c467 to 0x8a5c9b7 2024-09-16T13:17:48.738 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:48.739 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:48.742 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:48.744 INFO:teuthology.orchestra.run.smithi098.stdout:3392 trunc from 0x8a5c9b7 to 0x8368d97 2024-09-16T13:17:48.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:48.781 INFO:teuthology.orchestra.run.smithi098.stdout:3393 writesame 0xedd0ee5 thru 0xedd69ad data_size 0x3d(0x5ac9 bytes) 2024-09-16T13:17:48.781 INFO:teuthology.orchestra.run.smithi098.stdout:3394 read 0x4bc6886 thru 0x4bc6f4d (0x6c8 bytes) 2024-09-16T13:17:48.781 INFO:teuthology.orchestra.run.smithi098.stdout:3396 write 0xbb910d6 thru 0xbb9910a (0x8035 bytes) 2024-09-16T13:17:48.787 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:48.787 INFO:teuthology.orchestra.run.smithi098.stdout:3401 write 0x6ce5067 thru 0x6cedef7 (0x8e91 bytes) 2024-09-16T13:17:48.794 INFO:teuthology.orchestra.run.smithi098.stdout:3403 writesame 0xaaf4075 thru 0xaaf797f data_size 0x1(0x390b bytes) 2024-09-16T13:17:48.797 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:48.788+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:48.843 INFO:teuthology.orchestra.run.smithi098.stdout:3404 trunc from 0xedd69ae to 0x10d317d 2024-09-16T13:17:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:3405 write 0xff7ffd thru 0xff86bc (0x6c0 bytes) 2024-09-16T13:17:48.898 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:48.888+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:48.901 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:48.901 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:48.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:48.922 INFO:teuthology.orchestra.run.smithi098.stdout:3406 write 0xba31006 thru 0xba3a309 (0x9304 bytes) 2024-09-16T13:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout:3407 read 0x4249786 thru 0x424be83 (0x26fe bytes) 2024-09-16T13:17:48.928 INFO:teuthology.orchestra.run.smithi098.stdout:3408 write 0xa09a600 thru 0xa0a6b46 (0xc547 bytes) 2024-09-16T13:17:48.934 INFO:teuthology.orchestra.run.smithi098.stdout:3409 read 0x3a7677e thru 0x3a7afd3 (0x4856 bytes) 2024-09-16T13:17:48.950 INFO:teuthology.orchestra.run.smithi098.stdout:3411 write 0xea7e7e1 thru 0xea7f617 (0xe37 bytes) 2024-09-16T13:17:48.957 INFO:teuthology.orchestra.run.smithi098.stdout:3412 clone 21 order 18 su 32768 sc 10 2024-09-16T13:17:49.000 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:48.992+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:49.015 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:49.015 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:49.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:49.101 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:49.092+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:49.122 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:49.122 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:49.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:49.169 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone20 from 0xea7f618 (overlap 0x10d317d) to 0x35d6b2 2024-09-16T13:17:49.202 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:49.195+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:49.236 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:49.236 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:49.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:49.303 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:49.296+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:49.348 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:49.348 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:49.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:49.359 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #19, image image_client.0-clone19 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent20 2024-09-16T13:17:49.378 INFO:teuthology.orchestra.run.smithi098.stdout:3413 read 0x39bc9b6 thru 0x39c64bb (0x9b06 bytes) 2024-09-16T13:17:49.379 INFO:teuthology.orchestra.run.smithi098.stdout:3414 read 0x68def40 thru 0x68e8c5f (0x9d20 bytes) 2024-09-16T13:17:49.380 INFO:teuthology.orchestra.run.smithi098.stdout:3415 writesame 0x5eebcd3 thru 0x5ef533e data_size 0x6(0x966c bytes) 2024-09-16T13:17:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:49.395 INFO:teuthology.orchestra.run.smithi098.stdout:3417 punch from 0x6a46d07 to 0x6a4fe59, (0x9152 bytes) 2024-09-16T13:17:49.404 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:49.395+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:49.410 INFO:teuthology.orchestra.run.smithi098.stdout:3419 trunc from 0xea7f618 to 0x5089a0c 2024-09-16T13:17:49.449 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:49.455 INFO:teuthology.orchestra.run.smithi098.stdout:3421 trunc from 0x5089a0c to 0x861b11f 2024-09-16T13:17:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:49.465 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:49.467 INFO:teuthology.orchestra.run.smithi098.stdout:3422 write 0x5b24926 thru 0x5b2933c (0x4a17 bytes) 2024-09-16T13:17:49.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:49.477 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:49.477 INFO:teuthology.orchestra.run.smithi098.stdout:3424 write 0x6ae4db4 thru 0x6af13fe (0xc64b bytes) 2024-09-16T13:17:49.504 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:49.495+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:49.506 INFO:teuthology.orchestra.run.smithi098.stdout:3425 write 0xc0b607c thru 0xc0be0b1 (0x8036 bytes) 2024-09-16T13:17:49.514 INFO:teuthology.orchestra.run.smithi098.stdout:3426 write 0x105d2ca thru 0x10608a1 (0x35d8 bytes) 2024-09-16T13:17:49.524 INFO:teuthology.orchestra.run.smithi098.stdout:3428 write 0x8131cd0 thru 0x81403bd (0xe6ee bytes) 2024-09-16T13:17:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:49.534 INFO:teuthology.orchestra.run.smithi098.stdout:3431 read 0x850ddd3 thru 0x851607c (0x82aa bytes) 2024-09-16T13:17:49.536 INFO:teuthology.orchestra.run.smithi098.stdout:3435 trunc from 0xc0be0b2 to 0xace4e2c 2024-09-16T13:17:49.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:49.556 INFO:teuthology.orchestra.run.smithi098.stdout:3437 writesame 0x10cf0ef thru 0x10da8d2 data_size 0x6(0xb7e4 bytes) 2024-09-16T13:17:49.572 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:49.572 INFO:teuthology.orchestra.run.smithi098.stdout:3439 writesame 0x2b9d240 thru 0x2ba68bf data_size 0x38(0x9680 bytes) 2024-09-16T13:17:49.577 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:49.577 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:49.581 INFO:teuthology.orchestra.run.smithi098.stdout:3441 write 0x630e26d thru 0x631467f (0x6413 bytes) 2024-09-16T13:17:49.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:49.590 INFO:teuthology.orchestra.run.smithi098.stdout:3443 write 0x5db9203 thru 0x5dbb06d (0x1e6b bytes) 2024-09-16T13:17:49.595 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:49.595 INFO:teuthology.orchestra.run.smithi098.stdout:3445 read 0x354bbae thru 0x35584a3 (0xc8f6 bytes) 2024-09-16T13:17:49.595 INFO:teuthology.orchestra.run.smithi098.stdout:3446 writesame 0x3008360 thru 0x3012f43 data_size 0x1(0xabe4 bytes) 2024-09-16T13:17:49.604 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:49.596+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:49.640 INFO:teuthology.orchestra.run.smithi098.stdout:3447 punch from 0x68ca13b to 0x68cf9fb, (0x58c0 bytes) 2024-09-16T13:17:49.640 INFO:teuthology.orchestra.run.smithi098.stdout:3448 punch from 0x626883b to 0x626da74, (0x5239 bytes) 2024-09-16T13:17:49.640 INFO:teuthology.orchestra.run.smithi098.stdout:3449 write 0x86d18a1 thru 0x86dbf05 (0xa665 bytes) 2024-09-16T13:17:49.648 INFO:teuthology.orchestra.run.smithi098.stdout:3450 write 0x1f99fc3 thru 0x1fa28b0 (0x88ee bytes) 2024-09-16T13:17:49.667 INFO:teuthology.orchestra.run.smithi098.stdout:3453 write 0xccab7e0 thru 0xccb4a6c (0x928d bytes) 2024-09-16T13:17:49.675 INFO:teuthology.orchestra.run.smithi098.stdout:3454 punch from 0x777ce1e to 0x7780516, (0x36f8 bytes) 2024-09-16T13:17:49.675 INFO:teuthology.orchestra.run.smithi098.stdout:3455 punch from 0xb71fcb7 to 0xb72a3a5, (0xa6ee bytes) 2024-09-16T13:17:49.675 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:49.675 INFO:teuthology.orchestra.run.smithi098.stdout:3457 punch from 0x8199a2c to 0x81a3be7, (0xa1bb bytes) 2024-09-16T13:17:49.675 INFO:teuthology.orchestra.run.smithi098.stdout:3459 read 0x5390422 thru 0x53971fd (0x6ddc bytes) 2024-09-16T13:17:49.676 INFO:teuthology.orchestra.run.smithi098.stdout:3461 punch from 0x3d47704 to 0x3d4f77c, (0x8078 bytes) 2024-09-16T13:17:49.676 INFO:teuthology.orchestra.run.smithi098.stdout:3465 read 0x32357e8 thru 0x32402b0 (0xaac9 bytes) 2024-09-16T13:17:49.676 INFO:teuthology.orchestra.run.smithi098.stdout:3467 trunc from 0xccb4a6d to 0xbd56a65 2024-09-16T13:17:49.689 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:49.694 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:49.694 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:49.697 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:49.697 INFO:teuthology.orchestra.run.smithi098.stdout:3471 read 0xa4c32ed thru 0xa4c8c5b (0x596f bytes) 2024-09-16T13:17:49.697 INFO:teuthology.orchestra.run.smithi098.stdout:3473 read 0x87d85c3 thru 0x87e6838 (0xe276 bytes) 2024-09-16T13:17:49.698 INFO:teuthology.orchestra.run.smithi098.stdout:3475 write 0x4db7e86 thru 0x4dc701e (0xf199 bytes) 2024-09-16T13:17:49.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:49.706 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:49.699+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:49.708 INFO:teuthology.orchestra.run.smithi098.stdout:3476 punch from 0xbc51c86 to 0xbc5b219, (0x9593 bytes) 2024-09-16T13:17:49.708 INFO:teuthology.orchestra.run.smithi098.stdout:3477 writesame 0x361f929 thru 0x362c734 data_size 0x2(0xce0c bytes) 2024-09-16T13:17:49.711 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:49.728 INFO:teuthology.orchestra.run.smithi098.stdout:3478 punch from 0x2e22f20 to 0x2e29e2a, (0x6f0a bytes) 2024-09-16T13:17:49.728 INFO:teuthology.orchestra.run.smithi098.stdout:3479 write 0x9a39b29 thru 0x9a40750 (0x6c28 bytes) 2024-09-16T13:17:49.730 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:49.735 INFO:teuthology.orchestra.run.smithi098.stdout:3482 write 0x6c2004 thru 0x6c6c4c (0x4c49 bytes) 2024-09-16T13:17:49.742 INFO:teuthology.orchestra.run.smithi098.stdout:3483 writesame 0x782de93 thru 0x783294b data_size 0x25(0x4ab9 bytes) 2024-09-16T13:17:49.749 INFO:teuthology.orchestra.run.smithi098.stdout:3484 write 0x17df966 thru 0x17e4b42 (0x51dd bytes) 2024-09-16T13:17:49.755 INFO:teuthology.orchestra.run.smithi098.stdout:3485 punch from 0x1d0aa98 to 0x1d1617c, (0xb6e4 bytes) 2024-09-16T13:17:49.756 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:49.756 INFO:teuthology.orchestra.run.smithi098.stdout:3486 write 0xa40f5e thru 0xa4f270 (0xe313 bytes) 2024-09-16T13:17:49.762 INFO:teuthology.orchestra.run.smithi098.stdout:3487 write 0x154955a thru 0x1550748 (0x71ef bytes) 2024-09-16T13:17:49.776 INFO:teuthology.orchestra.run.smithi098.stdout:3490 trunc from 0xbd56a65 to 0x8895e1a 2024-09-16T13:17:49.776 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:49.795 INFO:teuthology.orchestra.run.smithi098.stdout:3491 write 0xc20aab thru 0xc2ec65 (0xe1bb bytes) 2024-09-16T13:17:49.798 INFO:teuthology.orchestra.run.smithi098.stdout:3493 punch from 0x802c1bd to 0x802ed58, (0x2b9b bytes) 2024-09-16T13:17:49.799 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:49.799 INFO:teuthology.orchestra.run.smithi098.stdout:3495 write 0x631fc66 thru 0x632e2ec (0xe687 bytes) 2024-09-16T13:17:49.807 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:49.799+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:49.808 INFO:teuthology.orchestra.run.smithi098.stdout:3497 read 0x78148a4 thru 0x781fc3b (0xb398 bytes) 2024-09-16T13:17:49.808 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:49.808 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:49.808 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:49.809 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:49.809 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:49.816 INFO:teuthology.orchestra.run.smithi098.stdout:3498 trunc from 0x8895e1a to 0x3b43eec 2024-09-16T13:17:49.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:49.829 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:49.854 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:49.861 INFO:teuthology.orchestra.run.smithi098.stdout:3501 punch from 0xc64d64 to 0xc6d278, (0x8514 bytes) 2024-09-16T13:17:49.861 INFO:teuthology.orchestra.run.smithi098.stdout:3502 read 0x21454b2 thru 0x2150670 (0xb1bf bytes) 2024-09-16T13:17:49.861 INFO:teuthology.orchestra.run.smithi098.stdout:3503 read 0x2eee7cf thru 0x2efb3cf (0xcc01 bytes) 2024-09-16T13:17:49.874 INFO:teuthology.orchestra.run.smithi098.stdout:3506 write 0x55b257e thru 0x55c21d4 (0xfc57 bytes) 2024-09-16T13:17:49.898 INFO:teuthology.orchestra.run.smithi098.stdout:3507 trunc from 0x55c21d5 to 0xdd7e53d 2024-09-16T13:17:49.908 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:49.899+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:49.910 INFO:teuthology.orchestra.run.smithi098.stdout:3508 write 0x63c8799 thru 0x63d1e03 (0x966b bytes) 2024-09-16T13:17:49.919 INFO:teuthology.orchestra.run.smithi098.stdout:3509 writesame 0xac9cca6 thru 0xaca8fc1 data_size 0x2(0xc31c bytes) 2024-09-16T13:17:49.933 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:49.933 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:6966 3.0 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 37.372961224999997, 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095509229999999994, 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.934 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 6966 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554531+0000", 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554532+0000", 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2000000000000001e-07 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554534+0000", 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9850000000000001e-06 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554537+0000", 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.664e-06 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.554541+0000", 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2909999999999999e-06 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557170+0000", 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026295310000000001 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557345+0000", 2024-09-16T13:17:49.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017473200000000001 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557618+0000", 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027282800000000002 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563504+0000", 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058862649999999999 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:49.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563923+0000", 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000419004 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564003+0000", 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0072000000000002e-05 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.564082+0000", 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9231000000000001e-05 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11223 3.8 e40/23 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head v 40'368, mlcod=39'365)", 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.460790140999997, 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01312405, 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466704+0000", 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.792e-06 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471144+0000", 2024-09-16T13:17:49.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044394830000000001 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471180+0000", 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5884999999999999e-05 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471194+0000", 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4258e-05 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471273+0000", 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9159999999999994e-05 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471305+0000", 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2236000000000001e-05 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479753+0000", 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0084474049999999998 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479826+0000", 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3831000000000005e-05 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11776 3.1a e40/23 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head v 40'72, mlcod=40'70)", 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.399893343999999, 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595333, 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:49.944 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527601+0000", 2024-09-16T13:17:49.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.522e-06 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536775+0000", 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091746569999999993 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536818+0000", 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2435000000000003e-05 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536828+0000", 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0624e-05 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536931+0000", 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010280000000000001 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536961+0000", 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0060999999999999e-05 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544146+0000", 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071846339999999996 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544195+0000", 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8600000000000002e-05 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.948 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13719 3.19 3:99fbe691:::rbd_data.112df1cc59bd.000000000000000c:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 26.646723773000002, 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011713708999999999, 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13719 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:49.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280771+0000", 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9240000000000001e-06 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280782+0000", 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1642e-05 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280806+0000", 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4119000000000001e-05 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280815+0000", 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7150000000000004e-06 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012396900000000001 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281142+0000", 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000203298 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281540+0000", 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039763500000000002 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.286598+0000", 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050579090000000002 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292380+0000", 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057823390000000001 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292433+0000", 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2612999999999999e-05 2024-09-16T13:17:49.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292482+0000", 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9546000000000003e-05 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13720 3.19 3:99acede8:::rbd_data.112df1cc59bd.000000000000000d:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 26.646660580999999, 2024-09-16T13:17:49.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011830879000000001, 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13720 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280836+0000", 2024-09-16T13:17:49.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.703e-06 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280844+0000", 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280849+0000", 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.254e-06 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282739+0000", 2024-09-16T13:17:49.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001889672 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282875+0000", 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013641 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.283217+0000", 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00034147099999999998 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:49.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.288528+0000", 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053110140000000002 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292558+0000", 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040304429999999999 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292620+0000", 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1588999999999996e-05 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292663+0000", 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2922e-05 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13726 3.19 3:9e45447e:::rbd_data.112df1cc59bd.0000000000000034:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 26.646559868000001, 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012171434, 2024-09-16T13:17:49.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13726 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280934+0000", 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9299999999999996e-07 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280950+0000", 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1141000000000001e-05 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280955+0000", 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5560000000000001e-06 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284230+0000", 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032752789999999999 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284286+0000", 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6456999999999998e-05 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284430+0000", 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014382899999999999 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.291179+0000", 2024-09-16T13:17:49.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067490099999999997 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293014+0000", 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018347719999999999 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293069+0000", 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5325999999999998e-05 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293104+0000", 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4892999999999999e-05 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14647 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 3801088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.292361930999999, 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0094151970000000001, 2024-09-16T13:17:49.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14647 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635131+0000", 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635132+0000", 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4300000000000001e-07 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635136+0000", 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3660000000000002e-06 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635150+0000", 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4491999999999999e-05 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635159+0000", 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2630000000000003e-06 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644405+0000", 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0092462220000000001 2024-09-16T13:17:49.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644476+0000", 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0958999999999996e-05 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644546+0000", 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9951999999999997e-05 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14648 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4325376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.292324244, 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096264239999999997, 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14648 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635168+0000", 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635169+0000", 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.071e-06 2024-09-16T13:17:49.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635173+0000", 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635188+0000", 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4389000000000001e-05 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635198+0000", 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0566e-05 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644579+0000", 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093805399999999997 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644670+0000", 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1240000000000001e-05 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644795+0000", 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012454 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14662 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1703936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.292078677999999, 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096453119999999996, 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.975 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14662 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635414+0000", 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635415+0000", 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5e-06 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635421+0000", 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730000000000001e-06 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635436+0000", 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4909000000000001e-05 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635450+0000", 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3183999999999999e-05 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644833+0000", 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093835959999999993 2024-09-16T13:17:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.644914+0000", 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0624000000000006e-05 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645059+0000", 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014542600000000001 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.979 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14663 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2228224~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.292033516, 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098295339999999991, 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14663 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635460+0000", 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1500000000000004e-07 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635463+0000", 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.478e-06 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635472+0000", 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4230000000000005e-06 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635498+0000", 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6543e-05 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645115+0000", 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096171710000000007 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645193+0000", 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7245999999999999e-05 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645289+0000", 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5958000000000005e-05 2024-09-16T13:17:49.983 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14664 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.291987026000001, 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010033297, 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.984 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14664 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635506+0000", 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5099999999999998e-07 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635511+0000", 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7820000000000001e-06 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635520+0000", 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5969999999999992e-06 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635529+0000", 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3249999999999997e-06 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645337+0000", 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098082589999999997 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645427+0000", 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9922999999999999e-05 2024-09-16T13:17:49.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645539+0000", 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011156 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14665 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.291957797999999, 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010434005999999999, 2024-09-16T13:17:49.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14665 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635538+0000", 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8329999999999998e-06 2024-09-16T13:17:49.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635546+0000", 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0760000000000003e-06 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635551+0000", 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6160000000000004e-06 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645677+0000", 2024-09-16T13:17:49.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010126546 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645821+0000", 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014406799999999999 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645969+0000", 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014730900000000001 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14678 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 0~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.291698354000001, 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012023611999999999, 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14678 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635797+0000", 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7039999999999999e-06 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635810+0000", 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3343000000000001e-05 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635845+0000", 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4751e-05 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635863+0000", 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8210999999999999e-05 2024-09-16T13:17:49.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646005+0000", 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010141766999999999 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646070+0000", 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5324e-05 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:49.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647818+0000", 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017475119999999999 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14679 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1835008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.291610647999999, 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012069985, 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:49.997 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14679 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:49.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635883+0000", 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3239999999999999e-06 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635889+0000", 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635905+0000", 2024-09-16T13:17:49.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6022999999999999e-05 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635911+0000", 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3740000000000003e-06 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647849+0000", 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011937906999999999 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647895+0000", 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5507999999999999e-05 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647952+0000", 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7132000000000001e-05 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14680 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2359296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.291574365999999, 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012172090999999999, 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14680 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.002 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:50.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635919+0000", 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8299999999999995e-07 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635922+0000", 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.027e-06 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635931+0000", 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6230000000000003e-06 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635936+0000", 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0069999999999997e-06 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647981+0000", 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012045273 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648033+0000", 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648090+0000", 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7154000000000002e-05 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14681 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4718592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:50.006 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.291551369, 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012312528, 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14681 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635942+0000", 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7599999999999996e-07 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635959+0000", 2024-09-16T13:17:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6954999999999999e-05 2024-09-16T13:17:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635968+0000", 2024-09-16T13:17:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8929999999999994e-06 2024-09-16T13:17:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635973+0000", 2024-09-16T13:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0170000000000004e-06 2024-09-16T13:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648120+0000", 2024-09-16T13:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012146847000000001 2024-09-16T13:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648170+0000", 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0368000000000003e-05 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648254+0000", 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3671999999999994e-05 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14694 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.291514294999999, 2024-09-16T13:17:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012792843999999999, 2024-09-16T13:17:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14694 2024-09-16T13:17:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635979+0000", 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0699999999999996e-07 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635982+0000", 2024-09-16T13:17:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8200000000000001e-06 2024-09-16T13:17:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635990+0000", 2024-09-16T13:17:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0239999999999997e-06 2024-09-16T13:17:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635995+0000", 2024-09-16T13:17:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648296+0000", 2024-09-16T13:17:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0123009 2024-09-16T13:17:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648397+0000", 2024-09-16T13:17:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101042 2024-09-16T13:17:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648771+0000", 2024-09-16T13:17:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037435099999999998 2024-09-16T13:17:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.017 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14695 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1966080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.291492277, 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012992136, 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.018 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14695 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636001+0000", 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0999999999999998e-07 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636004+0000", 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5959999999999998e-06 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636012+0000", 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:17:50.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636016+0000", 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9269999999999996e-06 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648804+0000", 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012787543 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648873+0000", 2024-09-16T13:17:50.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9263000000000004e-05 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648992+0000", 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119089 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14696 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2490368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.291470585999999, 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013132612, 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14696 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636023+0000", 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7300000000000005e-07 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636025+0000", 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.407e-06 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636033+0000", 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0129999999999993e-06 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636042+0000", 2024-09-16T13:17:50.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1540000000000008e-06 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649027+0000", 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012985096 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649089+0000", 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1641000000000002e-05 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649155+0000", 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5528e-05 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14697 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.291444684999998, 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013321177999999999, 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.027 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14697 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636049+0000", 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636051+0000", 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.57e-06 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636059+0000", 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9440000000000005e-06 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636067+0000", 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0460000000000006e-06 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649182+0000", 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01311522 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649245+0000", 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2930999999999998e-05 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649369+0000", 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123681 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.031 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:50.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:17:50.033 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:50.027+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:50.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:50.034 INFO:teuthology.orchestra.run.smithi098.stdout:3510 trunc from 0xdd7e53d to 0x1815887 2024-09-16T13:17:50.034 INFO:teuthology.orchestra.run.smithi098.stdout:3511 read 0x17c48a7 thru 0x17c90a9 (0x4803 bytes) 2024-09-16T13:17:50.035 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.044 INFO:teuthology.orchestra.run.smithi098.stdout:3513 write 0xb53d547 thru 0xb547633 (0xa0ed bytes) 2024-09-16T13:17:50.051 INFO:teuthology.orchestra.run.smithi098.stdout:3515 read 0x22bf050 thru 0x22c4c07 (0x5bb8 bytes) 2024-09-16T13:17:50.052 INFO:teuthology.orchestra.run.smithi098.stdout:3516 write 0x446a1ba thru 0x4477cc9 (0xdb10 bytes) 2024-09-16T13:17:50.074 INFO:teuthology.orchestra.run.smithi098.stdout:3517 trunc from 0xb547634 to 0x14b84d0 2024-09-16T13:17:50.111 INFO:teuthology.orchestra.run.smithi098.stdout:3518 writesame 0x76548e thru 0x76df37 data_size 0x1(0x8aaa bytes) 2024-09-16T13:17:50.133 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:50.124+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:50.164 INFO:teuthology.orchestra.run.smithi098.stdout:3520 trunc from 0x14b84d0 to 0xe1dec12 2024-09-16T13:17:50.176 INFO:teuthology.orchestra.run.smithi098.stdout:3521 read 0x19a96e8 thru 0x19b1d07 (0x8620 bytes) 2024-09-16T13:17:50.177 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.177 INFO:teuthology.orchestra.run.smithi098.stdout:3523 write 0x595f0ab thru 0x595fce4 (0xc3a bytes) 2024-09-16T13:17:50.179 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:50.179 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:50.182 INFO:teuthology.orchestra.run.smithi098.stdout:3524 writesame 0x6db7608 thru 0x6db83f2 data_size 0x1(0xdeb bytes) 2024-09-16T13:17:50.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:50.209 INFO:teuthology.orchestra.run.smithi098.stdout:3526 trunc from 0xe1dec12 to 0x4e1dbf2 2024-09-16T13:17:50.243 INFO:teuthology.orchestra.run.smithi098.stdout:3527 trunc from 0x4e1dbf2 to 0x5300e7 2024-09-16T13:17:50.244 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:50.236+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:50.284 INFO:teuthology.orchestra.run.smithi098.stdout:3529 punch from 0x3d683e to 0x3e21e1, (0xb9a3 bytes) 2024-09-16T13:17:50.284 INFO:teuthology.orchestra.run.smithi098.stdout:3530 read 0x27e212 thru 0x27f66e (0x145d bytes) 2024-09-16T13:17:50.284 INFO:teuthology.orchestra.run.smithi098.stdout:3531 punch from 0x15a649 to 0x16416f, (0x9b26 bytes) 2024-09-16T13:17:50.284 INFO:teuthology.orchestra.run.smithi098.stdout:3532 read 0x4b5367 thru 0x4bbbfc (0x6896 bytes) 2024-09-16T13:17:50.285 INFO:teuthology.orchestra.run.smithi098.stdout:3533 read 0x2a5f69 thru 0x2a6860 (0x8f8 bytes) 2024-09-16T13:17:50.285 INFO:teuthology.orchestra.run.smithi098.stdout:3534 punch from 0x29ef86 to 0x2acdeb, (0xde65 bytes) 2024-09-16T13:17:50.285 INFO:teuthology.orchestra.run.smithi098.stdout:3535 punch from 0x2f6a21 to 0x2fe789, (0x7d68 bytes) 2024-09-16T13:17:50.286 INFO:teuthology.orchestra.run.smithi098.stdout:3536 punch from 0x49386f to 0x49c97f, (0x9110 bytes) 2024-09-16T13:17:50.311 INFO:teuthology.orchestra.run.smithi098.stdout:3537 write 0xa1a4fa3 thru 0xa1b4d43 (0xfda1 bytes) 2024-09-16T13:17:50.314 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:50.315 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:50.315 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:50.321 INFO:teuthology.orchestra.run.smithi098.stdout:3538 read 0x30711aa thru 0x3073b9c (0x29f3 bytes) 2024-09-16T13:17:50.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:50.335 INFO:teuthology.orchestra.run.smithi098.stdout:3539 writesame 0xbe317f3 thru 0xbe33db1 data_size 0x1(0x25bf bytes) 2024-09-16T13:17:50.335 INFO:teuthology.orchestra.run.smithi098.stdout:3540 read 0x9d367a2 thru 0x9d3b6a5 (0x4f04 bytes) 2024-09-16T13:17:50.336 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:50.328+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:50.347 INFO:teuthology.orchestra.run.smithi098.stdout:3541 trunc from 0xbe33db2 to 0x4d8cdfa 2024-09-16T13:17:50.366 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.388 INFO:teuthology.orchestra.run.smithi098.stdout:3545 writesame 0xaf89a3d thru 0xaf94845 data_size 0x1(0xae09 bytes) 2024-09-16T13:17:50.389 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.389 INFO:teuthology.orchestra.run.smithi098.stdout:3547 read 0x10cf759 thru 0x10d06f0 (0xf98 bytes) 2024-09-16T13:17:50.389 INFO:teuthology.orchestra.run.smithi098.stdout:3548 write 0x8642012 thru 0x864d6a3 (0xb692 bytes) 2024-09-16T13:17:50.397 INFO:teuthology.orchestra.run.smithi098.stdout:3550 write 0xa3cbf40 thru 0xa3d7a04 (0xbac5 bytes) 2024-09-16T13:17:50.404 INFO:teuthology.orchestra.run.smithi098.stdout:3551 write 0x536cbb5 thru 0x53767e1 (0x9c2d bytes) 2024-09-16T13:17:50.409 INFO:teuthology.orchestra.run.smithi098.stdout:3552 punch from 0x918957b to 0x9190359, (0x6dde bytes) 2024-09-16T13:17:50.424 INFO:teuthology.orchestra.run.smithi098.stdout:3553 write 0xcac542b thru 0xcacc113 (0x6ce9 bytes) 2024-09-16T13:17:50.432 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.432 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.432 INFO:teuthology.orchestra.run.smithi098.stdout:3556 trunc from 0xcacc114 to 0xc941421 2024-09-16T13:17:50.437 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:50.427+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:50.443 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:17:50.443 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.431235497999999, 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:17:50.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:17:50.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.430449998999997, 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:17:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.429283077000001, 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:17:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:17:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.426408446000003, 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:17:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.423394706000003, 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:17:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.398190503000002, 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:17:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 45.632041614000002, 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:17:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5115 3.4 e34/23 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head v 34'14, mlcod=31'12)", 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.646993508000001, 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029400680000000001, 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791635+0000", 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4200000000000005e-07 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791643+0000", 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.157e-06 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791660+0000", 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6246e-05 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791666+0000", 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791743+0000", 2024-09-16T13:17:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7000000000000001e-05 2024-09-16T13:17:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791756+0000", 2024-09-16T13:17:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2729e-05 2024-09-16T13:17:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794523+0000", 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027675949999999999 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794575+0000", 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1082000000000003e-05 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.379939311000001, 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:17:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:17:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.379329871000003, 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:17:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6963 3.4 e37/23 3:2713f365:::rbd_data.112d6fb8d387.0000000000000133:head v 37'31, mlcod=37'29)", 2024-09-16T13:17:50.488 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 37.882765720999998, 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027598190000000002, 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555862+0000", 2024-09-16T13:17:50.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555864+0000", 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555882+0000", 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7816e-05 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555909+0000", 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7838999999999999e-05 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.555923+0000", 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3577000000000001e-05 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556083+0000", 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015987299999999999 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556132+0000", 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8655000000000001e-05 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558539+0000", 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0024076969999999999 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558622+0000", 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2777999999999999e-05 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.492 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 37.882046056999997, 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:17:50.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 37.880945240999999, 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:17:50.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:17:50.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 37.880805600999999, 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:17:50.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:17:50.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 37.880305071000002, 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.506 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:17:50.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.885135685000002, 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:17:50.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 22.628721844000001, 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.519 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 22.016158737000001, 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:22854 3.9 e57/23 3:924a2914:::rbd_data.112db5030c67.0000000000000008:head v 57'198, mlcod=57'196)", 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 8.8976381730000007, 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029464980000000001, 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:17:50.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540995+0000", 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541018+0000", 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2382999999999999e-05 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541079+0000", 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1773999999999995e-05 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541094+0000", 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4414e-05 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.542159+0000", 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010653749999999999 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.542208+0000", 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8517000000000003e-05 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.543797+0000", 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015893210000000001 2024-09-16T13:17:50.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.543936+0000", 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000139367 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:25483 3.0 e64/23 3:0369a05e:::rbd_data.112de78841b1.00000000000000ee:head v 64'159, mlcod=64'157)", 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:49.846684+0000", 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.59194402099999999, 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028849119999999999, 2024-09-16T13:17:50.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846684+0000", 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846684+0000", 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846686+0000", 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5850000000000001e-06 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846695+0000", 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5719999999999992e-06 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846717+0000", 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1946999999999998e-05 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846726+0000", 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4800000000000001e-06 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846798+0000", 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2819999999999997e-05 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846830+0000", 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1461000000000003e-05 2024-09-16T13:17:50.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.849512+0000", 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026821039999999998 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.849569+0000", 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6943e-05 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.532 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:50.533 INFO:teuthology.orchestra.run.smithi098.stdout:3557 write 0x46d8d20 thru 0x46e7561 (0xe842 bytes) 2024-09-16T13:17:50.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:50.535 INFO:teuthology.orchestra.run.smithi098.stdout:3560 writesame 0x3da93de thru 0x3db6a0a data_size 0x1(0xd62d bytes) 2024-09-16T13:17:50.535 INFO:teuthology.orchestra.run.smithi098.stdout:3561 read 0x434cd7 thru 0x43cbf9 (0x7f23 bytes) 2024-09-16T13:17:50.535 INFO:teuthology.orchestra.run.smithi098.stdout:3562 write 0x1137ed thru 0x11633f (0x2b53 bytes) 2024-09-16T13:17:50.535 INFO:teuthology.orchestra.run.smithi098.stdout:3564 write 0xc156ed thru 0xc1fe4e (0xa762 bytes) 2024-09-16T13:17:50.535 INFO:teuthology.orchestra.run.smithi098.stdout:3565 writesame 0x17bdc00 thru 0x17c044e data_size 0x1(0x284f bytes) 2024-09-16T13:17:50.542 INFO:teuthology.orchestra.run.smithi098.stdout:3566 trunc from 0xc941421 to 0xe695b62 2024-09-16T13:17:50.542 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:50.532+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:50.567 INFO:teuthology.orchestra.run.smithi098.stdout:3567 write 0xeba42db thru 0xeba8be5 (0x490b bytes) 2024-09-16T13:17:50.576 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.590 INFO:teuthology.orchestra.run.smithi098.stdout:3569 trunc from 0xeba8be6 to 0x726e337 2024-09-16T13:17:50.621 INFO:teuthology.orchestra.run.smithi098.stdout:3572 trunc from 0x726e337 to 0x9a9dbb7 2024-09-16T13:17:50.632 INFO:teuthology.orchestra.run.smithi098.stdout:3573 read 0x2df1c0 thru 0x2e8fad (0x9dee bytes) 2024-09-16T13:17:50.638 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:50.631+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:50.649 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:50.649 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:50.649 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:50.649 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:50.649 INFO:teuthology.orchestra.run.smithi098.stdout:3575 trunc from 0x9a9dbb7 to 0x111c11 2024-09-16T13:17:50.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:50.705 INFO:teuthology.orchestra.run.smithi098.stdout:3576 trunc from 0x111c11 to 0x8596821 2024-09-16T13:17:50.724 INFO:teuthology.orchestra.run.smithi098.stdout:3577 trunc from 0x8596821 to 0x41c1f0c 2024-09-16T13:17:50.739 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:50.728+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:50.756 INFO:teuthology.orchestra.run.smithi098.stdout:3578 trunc from 0x41c1f0c to 0xdd0f922 2024-09-16T13:17:50.770 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.770 INFO:teuthology.orchestra.run.smithi098.stdout:3581 write 0xb62b52 thru 0xb6f845 (0xccf4 bytes) 2024-09-16T13:17:50.779 INFO:teuthology.orchestra.run.smithi098.stdout:3582 read 0xcf23962 thru 0xcf25f0c (0x25ab bytes) 2024-09-16T13:17:50.779 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.780 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.780 INFO:teuthology.orchestra.run.smithi098.stdout:3589 write 0x8589a3f thru 0x858c93f (0x2f01 bytes) 2024-09-16T13:17:50.785 INFO:teuthology.orchestra.run.smithi098.stdout:3590 read 0x92c290a thru 0x92cf54c (0xcc43 bytes) 2024-09-16T13:17:50.785 INFO:teuthology.orchestra.run.smithi098.stdout:3591 punch from 0xc8e1374 to 0xc8e5840, (0x44cc bytes) 2024-09-16T13:17:50.785 INFO:teuthology.orchestra.run.smithi098.stdout:3593 read 0x4f72d9e thru 0x4f77adc (0x4d3f bytes) 2024-09-16T13:17:50.785 INFO:teuthology.orchestra.run.smithi098.stdout:3594 punch from 0x2e9d87 to 0x2f3e4e, (0xa0c7 bytes) 2024-09-16T13:17:50.785 INFO:teuthology.orchestra.run.smithi098.stdout:3595 writesame 0x2db3208 thru 0x2dbabe3 data_size 0x16(0x79dc bytes) 2024-09-16T13:17:50.794 INFO:teuthology.orchestra.run.smithi098.stdout:3599 read 0x54fab42 thru 0x550826a (0xd729 bytes) 2024-09-16T13:17:50.794 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.794 INFO:teuthology.orchestra.run.smithi098.stdout:3601 read 0xc12c85b thru 0xc134b42 (0x82e8 bytes) 2024-09-16T13:17:50.794 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:50.806 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:50.806 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:50.806 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:50.806 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:50.807 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:50.807 INFO:teuthology.orchestra.run.smithi098.stdout:3603 trunc from 0xdd0f922 to 0x58f47e7 2024-09-16T13:17:50.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:50.828 INFO:teuthology.orchestra.run.smithi098.stdout:3605 write 0x18404f1 thru 0x184bff0 (0xbb00 bytes) 2024-09-16T13:17:50.839 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:50.831+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:50.855 INFO:teuthology.orchestra.run.smithi098.stdout:3606 write 0xa8da820 thru 0xa8e5e13 (0xb5f4 bytes) 2024-09-16T13:17:50.881 INFO:teuthology.orchestra.run.smithi098.stdout:3607 trunc from 0xa8e5e14 to 0xb3384b 2024-09-16T13:17:50.906 INFO:teuthology.orchestra.run.smithi098.stdout:3609 trunc from 0xb3384b to 0x10022b8 2024-09-16T13:17:50.920 INFO:teuthology.orchestra.run.smithi098.stdout:3611 trunc from 0x10022b8 to 0x2076fb8 2024-09-16T13:17:50.926 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:50.926 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:50.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:50.926 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:50.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.926 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5101 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.926 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:50.926 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.142806094999997, 2024-09-16T13:17:50.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595248, 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5101 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777333+0000", 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1949999999999996e-06 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777342+0000", 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6950000000000006e-06 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777348+0000", 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785241+0000", 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078929329999999995 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785391+0000", 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150113 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785969+0000", 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057822899999999998 2024-09-16T13:17:50.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789745+0000", 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037756529999999999 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793823+0000", 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040786299999999998 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793880+0000", 2024-09-16T13:17:50.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7157999999999997e-05 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793924+0000", 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3407e-05 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5102 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:50.932 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.142758803, 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016663710000000002, 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5102 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777377+0000", 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4600000000000003e-07 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777381+0000", 2024-09-16T13:17:50.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5290000000000002e-06 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777393+0000", 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2001000000000001e-05 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777401+0000", 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3869999999999997e-06 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786753+0000", 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093523649999999996 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786860+0000", 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106921 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787286+0000", 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042651800000000003 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793815+0000", 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065290579999999999 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793932+0000", 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116688 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794002+0000", 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0313000000000003e-05 2024-09-16T13:17:50.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794040+0000", 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7083999999999999e-05 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5103 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.142697917, 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017505544000000001, 2024-09-16T13:17:50.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5103 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777438+0000", 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.78e-07 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777442+0000", 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5950000000000001e-06 2024-09-16T13:17:50.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777453+0000", 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0372e-05 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777458+0000", 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6559999999999997e-06 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788034+0000", 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010575753 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788111+0000", 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7038000000000003e-05 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788463+0000", 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000352028 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794263+0000", 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005800342 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794837+0000", 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057334300000000003 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794897+0000", 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0690000000000003e-05 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794942+0000", 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4848999999999997e-05 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5104 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.142653508999999, 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017694142999999999, 2024-09-16T13:17:50.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5104 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777482+0000", 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777487+0000", 2024-09-16T13:17:50.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6639999999999997e-06 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777496+0000", 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2890000000000003e-06 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777501+0000", 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2730000000000002e-06 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788951+0000", 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011449429000000001 2024-09-16T13:17:50.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789007+0000", 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5643999999999998e-05 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789205+0000", 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019820800000000001 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794268+0000", 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050636850000000001 2024-09-16T13:17:50.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795094+0000", 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082569700000000002 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795139+0000", 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5055000000000001e-05 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795175+0000", 2024-09-16T13:17:50.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6149000000000002e-05 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5105 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.142591168000003, 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017729482000000001, 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5105 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777544+0000", 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1100000000000004e-07 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777549+0000", 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777561+0000", 2024-09-16T13:17:50.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1524e-05 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777566+0000", 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.558e-06 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789781+0000", 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012214981 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789870+0000", 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9171000000000004e-05 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790313+0000", 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044255299999999999 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794273+0000", 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039596129999999998 2024-09-16T13:17:50.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795213+0000", 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094086099999999998 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795239+0000", 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5324e-05 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795273+0000", 2024-09-16T13:17:50.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4198e-05 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5106 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.142546023999998, 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018621867, 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.956 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5106 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:17:50.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777590+0000", 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777594+0000", 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6750000000000001e-06 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777604+0000", 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0076e-05 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777610+0000", 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7529999999999998e-06 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790893+0000", 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013282845 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790999+0000", 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106289 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791493+0000", 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049357800000000005 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794662+0000", 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031691409999999999 2024-09-16T13:17:50.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796146+0000", 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00148375 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796176+0000", 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0349e-05 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796211+0000", 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4530000000000003e-05 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5107 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.142514763000001, 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018745067000000001, 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.962 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5107 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777621+0000", 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-07 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777624+0000", 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7950000000000001e-06 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777633+0000", 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5979999999999997e-06 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777638+0000", 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.989e-06 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792024+0000", 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014385692 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792104+0000", 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0567000000000003e-05 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792340+0000", 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023624200000000001 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794667+0000", 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023269990000000002 2024-09-16T13:17:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796304+0000", 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001636981 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796331+0000", 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6492999999999999e-05 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796365+0000", 2024-09-16T13:17:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4102e-05 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5108 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.142460933999999, 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019090617000000001, 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.968 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5108 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777677+0000", 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7879999999999998e-06 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777686+0000", 2024-09-16T13:17:50.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7679999999999997e-06 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777689+0000", 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.123e-06 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792751+0000", 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01506158 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792855+0000", 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010456 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793305+0000", 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045006100000000003 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794671+0000", 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001366304 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796393+0000", 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001721272 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796733+0000", 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000340611 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796764+0000", 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.099e-05 2024-09-16T13:17:50.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5208 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.040701482999999, 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014882177999999999, 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.974 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5208 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879434+0000", 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.322e-06 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879443+0000", 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879459+0000", 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6133999999999998e-05 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879467+0000", 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2400000000000007e-06 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888339+0000", 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088721619999999994 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888461+0000", 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121341 2024-09-16T13:17:50.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888819+0000", 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035782799999999998 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891271+0000", 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002452872 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894224+0000", 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029522910000000001 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894275+0000", 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1647999999999997e-05 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894315+0000", 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9993999999999997e-05 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.980 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5210 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.980 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:50.980 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.040146295, 2024-09-16T13:17:50.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01466423, 2024-09-16T13:17:50.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.980 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5210 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:17:50.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879989+0000", 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8800000000000001e-07 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879995+0000", 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880017+0000", 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2101000000000001e-05 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880024+0000", 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9240000000000001e-06 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890250+0000", 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010226012 2024-09-16T13:17:50.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890344+0000", 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.446e-05 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890770+0000", 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042575099999999998 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894344+0000", 2024-09-16T13:17:50.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003574163 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894549+0000", 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020477599999999999 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894606+0000", 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6891000000000001e-05 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894653+0000", 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6867999999999999e-05 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5211 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.040102232000002, 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014800604, 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.986 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5211 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:17:50.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880033+0000", 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.06e-07 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880039+0000", 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880052+0000", 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-05 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880059+0000", 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3090000000000001e-06 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891316+0000", 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011257118999999999 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891374+0000", 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8020999999999999e-05 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891577+0000", 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020371699999999999 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894348+0000", 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027708300000000002 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894726+0000", 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037819399999999999 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894784+0000", 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7667999999999997e-05 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894833+0000", 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8936e-05 2024-09-16T13:17:50.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5212 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.039746858000001, 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014645704000000001, 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.992 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5212 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880389+0000", 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1680000000000001e-06 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880395+0000", 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880408+0000", 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.308e-05 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880414+0000", 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.105e-06 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892037+0000", 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011622402 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892116+0000", 2024-09-16T13:17:50.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9939999999999997e-05 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892597+0000", 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048076500000000002 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894492+0000", 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001895246 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:50.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894921+0000", 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000428114 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894985+0000", 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4777999999999995e-05 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895034+0000", 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8133999999999997e-05 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:50.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5213 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.039613707999997, 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014632608, 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5213 2024-09-16T13:17:50.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:50.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880522+0000", 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2600000000000001e-07 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880527+0000", 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880538+0000", 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917999999999999e-05 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880543+0000", 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.361e-06 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893069+0000", 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012525497999999999 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893152+0000", 2024-09-16T13:17:51.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2805000000000004e-05 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893635+0000", 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048315100000000002 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894852+0000", 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001216806 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895075+0000", 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022300299999999999 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895122+0000", 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7077999999999999e-05 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895154+0000", 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1919999999999999e-05 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.003 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5214 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.039459057000002, 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017785977000000001, 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:51.004 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5214 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880681+0000", 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880689+0000", 2024-09-16T13:17:51.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3610000000000001e-06 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880694+0000", 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5809999999999998e-06 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895186+0000", 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014492430000000001 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:51.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895311+0000", 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012518300000000001 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895760+0000", 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044864900000000002 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898337+0000", 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025765229999999998 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898354+0000", 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7538999999999999e-05 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898406+0000", 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2054e-05 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898462+0000", 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5404e-05 2024-09-16T13:17:51.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5215 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.039194223000003, 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018191319000000001, 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:51.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5215 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880941+0000", 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880946+0000", 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3710000000000002e-06 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:51.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880955+0000", 2024-09-16T13:17:51.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1230000000000006e-06 2024-09-16T13:17:51.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880959+0000", 2024-09-16T13:17:51.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:17:51.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896782+0000", 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015822716000000001 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896876+0000", 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3667999999999998e-05 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.897289+0000", 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041357399999999999 2024-09-16T13:17:51.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898474+0000", 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011847209999999999 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899036+0000", 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056215699999999998 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899090+0000", 2024-09-16T13:17:51.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3662999999999998e-05 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899132+0000", 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1814000000000003e-05 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5282 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:51.016 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.970522109000001, 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014924988, 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5282 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.53e-07 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949635+0000", 2024-09-16T13:17:51.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2422999999999999e-05 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949641+0000", 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.896e-06 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949644+0000", 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5270000000000001e-06 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:51.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959656+0000", 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010012378000000001 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959747+0000", 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1273000000000005e-05 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960045+0000", 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029733 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963196+0000", 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031512620000000002 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964453+0000", 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001257357 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964502+0000", 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9044e-05 2024-09-16T13:17:51.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:51.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964538+0000", 2024-09-16T13:17:51.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5244999999999999e-05 2024-09-16T13:17:51.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.022 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:51.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.022 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5283 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.970486954999998, 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015065762, 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5283 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:51.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949650+0000", 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.736e-06 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949653+0000", 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2969999999999999e-06 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949655+0000", 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1890000000000001e-06 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960643+0000", 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010988015 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960732+0000", 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9097000000000004e-05 2024-09-16T13:17:51.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961195+0000", 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000462292 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963208+0000", 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002013641 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964585+0000", 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013765769999999999 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964669+0000", 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3863000000000007e-05 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964713+0000", 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4706999999999997e-05 2024-09-16T13:17:51.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5284 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.970477086000002, 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015152802, 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:51.029 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5284 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5899999999999998e-07 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949659+0000", 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.601e-06 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949672+0000", 2024-09-16T13:17:51.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2364999999999999e-05 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949674+0000", 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961869+0000", 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012194903999999999 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961971+0000", 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102154 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.962502+0000", 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00053037500000000003 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964751+0000", 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022487800000000001 2024-09-16T13:17:51.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964755+0000", 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964789+0000", 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4054000000000003e-05 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964810+0000", 2024-09-16T13:17:51.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0922000000000001e-05 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5285 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.970458168, 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015553022, 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:51.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5285 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949677+0000", 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6899999999999999e-07 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949679+0000", 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0090000000000002e-06 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949682+0000", 2024-09-16T13:17:51.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2550000000000002e-06 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949685+0000", 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.621e-06 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963409+0000", 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013724485999999999 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963497+0000", 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7560999999999997e-05 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963933+0000", 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043592700000000002 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964820+0000", 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088772299999999998 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965165+0000", 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000344426 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965201+0000", 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5967000000000003e-05 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965230+0000", 2024-09-16T13:17:51.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11223 3.8 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 33927~20122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.459285838, 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019333487, 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:51.041 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11223 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:51.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460851+0000", 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5730000000000002e-06 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464824+0000", 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039725730000000001 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464861+0000", 2024-09-16T13:17:51.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7233000000000001e-05 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464872+0000", 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209000000000001e-05 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464987+0000", 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011423 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465188+0000", 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020091600000000001 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465929+0000", 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00074102 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.475697+0000", 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009768367 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480068+0000", 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043713290000000002 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480143+0000", 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5000999999999996e-05 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480182+0000", 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9035999999999997e-05 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.046 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:51.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:51.047 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:51.047 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:51.048 INFO:teuthology.orchestra.run.smithi098.stdout:3612 punch from 0x1301cc4 to 0x1302f3a, (0x1276 bytes) 2024-09-16T13:17:51.048 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:51.050 INFO:teuthology.orchestra.run.smithi098.stdout:3614 trunc from 0x2076fb8 to 0xe9985c5 2024-09-16T13:17:51.050 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.050 INFO:teuthology.orchestra.run.smithi098.stdout:3616 punch from 0xa345c70 to 0xa34d00b, (0x739b bytes) 2024-09-16T13:17:51.050 INFO:teuthology.orchestra.run.smithi098.stdout:3617 trunc from 0xe9985c5 to 0x49c575 2024-09-16T13:17:51.050 INFO:teuthology.orchestra.run.smithi098.stdout:3618 read 0x3759fc thru 0x3764c2 (0xac7 bytes) 2024-09-16T13:17:51.050 INFO:teuthology.orchestra.run.smithi098.stdout:3621 read 0x22ae29 thru 0x23722d (0xc405 bytes) 2024-09-16T13:17:51.050 INFO:teuthology.orchestra.run.smithi098.stdout:3622 read 0x24f38 thru 0x2f886 (0xa94f bytes) 2024-09-16T13:17:51.050 INFO:teuthology.orchestra.run.smithi098.stdout:3625 punch from 0x387562 to 0x39342e, (0xbecc bytes) 2024-09-16T13:17:51.051 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.051 INFO:teuthology.orchestra.run.smithi098.stdout:3630 trunc from 0x49c575 to 0x3316df0 2024-09-16T13:17:51.051 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:17:51.052 INFO:teuthology.orchestra.run.smithi098.stdout:3631 write 0x1ef8f28 thru 0x1effa83 (0x6b5c bytes) 2024-09-16T13:17:51.052 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:51.040+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:51.084 INFO:teuthology.orchestra.run.smithi098.stdout:3633 write 0xde4d58d thru 0xde58b4e (0xb5c2 bytes) 2024-09-16T13:17:51.094 INFO:teuthology.orchestra.run.smithi098.stdout:3634 write 0x181c239 thru 0x1821ec5 (0x5c8d bytes) 2024-09-16T13:17:51.104 INFO:teuthology.orchestra.run.smithi098.stdout:3635 write 0x82618e0 thru 0x826ed4d (0xd46e bytes) 2024-09-16T13:17:51.112 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.113 INFO:teuthology.orchestra.run.smithi098.stdout:3640 read 0xd1bfc99 thru 0xd1c258d (0x28f5 bytes) 2024-09-16T13:17:51.113 INFO:teuthology.orchestra.run.smithi098.stdout:3642 write 0xde0f48c thru 0xde11907 (0x247c bytes) 2024-09-16T13:17:51.118 INFO:teuthology.orchestra.run.smithi098.stdout:3643 read 0x93069e2 thru 0x930808b (0x16aa bytes) 2024-09-16T13:17:51.119 INFO:teuthology.orchestra.run.smithi098.stdout:3644 write 0xd840eeb thru 0xd850048 (0xf15e bytes) 2024-09-16T13:17:51.128 INFO:teuthology.orchestra.run.smithi098.stdout:3647 writesame 0x82576eb thru 0x826144a data_size 0x10(0x9d60 bytes) 2024-09-16T13:17:51.138 INFO:teuthology.orchestra.run.smithi098.stdout:3648 punch from 0x8c591fa to 0x8c6028d, (0x7093 bytes) 2024-09-16T13:17:51.138 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.138 INFO:teuthology.orchestra.run.smithi098.stdout:3652 read 0xdd4f47f thru 0xdd56392 (0x6f14 bytes) 2024-09-16T13:17:51.138 INFO:teuthology.orchestra.run.smithi098.stdout:3653 write 0x23e702e thru 0x23f218c (0xb15f bytes) 2024-09-16T13:17:51.146 INFO:teuthology.orchestra.run.smithi098.stdout:3654 read 0x3b35377 thru 0x3b3a230 (0x4eba bytes) 2024-09-16T13:17:51.147 INFO:teuthology.orchestra.run.smithi098.stdout:3656 write 0x4838ba7 thru 0x4843417 (0xa871 bytes) 2024-09-16T13:17:51.149 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:51.143+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:51.166 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:51.167 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:51.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:51.184 INFO:teuthology.orchestra.run.smithi098.stdout:3657 trunc from 0xde58b4f to 0xa7ca92 2024-09-16T13:17:51.217 INFO:teuthology.orchestra.run.smithi098.stdout:3658 write 0x23db66f thru 0x23dd5c8 (0x1f5a bytes) 2024-09-16T13:17:51.244 INFO:teuthology.orchestra.run.smithi098.stdout:3659 write 0x61a760b thru 0x61b0f2b (0x9921 bytes) 2024-09-16T13:17:51.250 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:51.243+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:51.256 INFO:teuthology.orchestra.run.smithi098.stdout:3660 read 0x480f72a thru 0x4810806 (0x10dd bytes) 2024-09-16T13:17:51.256 INFO:teuthology.orchestra.run.smithi098.stdout:3661 read 0x5e2b5a9 thru 0x5e36cf1 (0xb749 bytes) 2024-09-16T13:17:51.257 INFO:teuthology.orchestra.run.smithi098.stdout:3662 write 0x41082eb thru 0x411796a (0xf680 bytes) 2024-09-16T13:17:51.264 INFO:teuthology.orchestra.run.smithi098.stdout:3665 read 0x48f61ec thru 0x4901aac (0xb8c1 bytes) 2024-09-16T13:17:51.264 INFO:teuthology.orchestra.run.smithi098.stdout:3666 read 0x503d421 thru 0x504bd4b (0xe92b bytes) 2024-09-16T13:17:51.264 INFO:teuthology.orchestra.run.smithi098.stdout:3667 read 0x2d29fbf thru 0x2d338bb (0x98fd bytes) 2024-09-16T13:17:51.264 INFO:teuthology.orchestra.run.smithi098.stdout:3670 read 0x58177d5 thru 0x581d4c6 (0x5cf2 bytes) 2024-09-16T13:17:51.265 INFO:teuthology.orchestra.run.smithi098.stdout:3671 punch from 0x5f20a60 to 0x5f272d3, (0x6873 bytes) 2024-09-16T13:17:51.265 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.278 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:51.278 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:51.278 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:51.279 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:51.279 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:51.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:51.291 INFO:teuthology.orchestra.run.smithi098.stdout:3673 write 0xce25387 thru 0xce29fb0 (0x4c2a bytes) 2024-09-16T13:17:51.300 INFO:teuthology.orchestra.run.smithi098.stdout:3674 read 0x216e11c thru 0x2173aa3 (0x5988 bytes) 2024-09-16T13:17:51.300 INFO:teuthology.orchestra.run.smithi098.stdout:3675 trunc from 0xce29fb1 to 0xd94c1d6 2024-09-16T13:17:51.311 INFO:teuthology.orchestra.run.smithi098.stdout:3677 writesame 0x915b9d5 thru 0x9166495 data_size 0x9(0xaac1 bytes) 2024-09-16T13:17:51.323 INFO:teuthology.orchestra.run.smithi098.stdout:3678 write 0x9cd9ebe thru 0x9ce6b9c (0xccdf bytes) 2024-09-16T13:17:51.331 INFO:teuthology.orchestra.run.smithi098.stdout:3679 read 0xa218bf0 thru 0xa21d719 (0x4b2a bytes) 2024-09-16T13:17:51.332 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.332 INFO:teuthology.orchestra.run.smithi098.stdout:3682 writesame 0x34bac2c thru 0x34bc33c data_size 0x1(0x1711 bytes) 2024-09-16T13:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout:3683 read 0x6f2d2c8 thru 0x6f35ad6 (0x880f bytes) 2024-09-16T13:17:51.343 INFO:teuthology.orchestra.run.smithi098.stdout:3684 writesame 0x1347401 thru 0x13565e2 data_size 0x7(0xf1e2 bytes) 2024-09-16T13:17:51.350 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:51.340+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:51.357 INFO:teuthology.orchestra.run.smithi098.stdout:3686 read 0x55e3f57 thru 0x55ea0f1 (0x619b bytes) 2024-09-16T13:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout:3687 punch from 0x95fe461 to 0x9602dee, (0x498d bytes) 2024-09-16T13:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout:3688 punch from 0xcba7adb to 0xcbaed3d, (0x7262 bytes) 2024-09-16T13:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout:3690 read 0x4fe2665 thru 0x4fe9981 (0x731d bytes) 2024-09-16T13:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.358 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.359 INFO:teuthology.orchestra.run.smithi098.stdout:3696 read 0x663d259 thru 0x664b099 (0xde41 bytes) 2024-09-16T13:17:51.371 INFO:teuthology.orchestra.run.smithi098.stdout:3697 writesame 0xda995d4 thru 0xda9ae8f data_size 0x1(0x18bc bytes) 2024-09-16T13:17:51.371 INFO:teuthology.orchestra.run.smithi098.stdout:3698 read 0xa943a0e thru 0xa94e60b (0xabfe bytes) 2024-09-16T13:17:51.371 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.371 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.371 INFO:teuthology.orchestra.run.smithi098.stdout:3701 read 0x1eb086a thru 0x1eb1253 (0x9ea bytes) 2024-09-16T13:17:51.371 INFO:teuthology.orchestra.run.smithi098.stdout:3702 writesame 0x8607e33 thru 0x8615f02 data_size 0x84(0xe0d0 bytes) 2024-09-16T13:17:51.383 INFO:teuthology.orchestra.run.smithi098.stdout:3703 read 0x5377ce3 thru 0x53845fe (0xc91c bytes) 2024-09-16T13:17:51.383 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.383 INFO:teuthology.orchestra.run.smithi098.stdout:3705 writesame 0x89db74f thru 0x89e5b32 data_size 0x22(0xa3e4 bytes) 2024-09-16T13:17:51.393 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:51.393 INFO:teuthology.orchestra.run.smithi098.stdout:3708 clone 22 order 24 su 262144 sc 8 2024-09-16T13:17:51.437 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:51.437 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:51.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:51.437 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:51.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8369 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.374540259, 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020058919000000001, 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8369 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056335+0000", 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056338+0000", 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056344+0000", 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8150000000000002e-06 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056348+0000", 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:17:51.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.070951+0000", 2024-09-16T13:17:51.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014603055 2024-09-16T13:17:51.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071060+0000", 2024-09-16T13:17:51.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109462 2024-09-16T13:17:51.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071508+0000", 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044824200000000001 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074304+0000", 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027951719999999998 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076331+0000", 2024-09-16T13:17:51.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020276529999999999 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076369+0000", 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7557e-05 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076393+0000", 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4445999999999999e-05 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8370 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.374522941000002, 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021240851000000002, 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8370 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-16T13:17:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056355+0000", 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.667e-06 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056360+0000", 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6620000000000002e-06 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056366+0000", 2024-09-16T13:17:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072672+0000", 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016306157000000002 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072746+0000", 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3648999999999999e-05 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.073364+0000", 2024-09-16T13:17:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061885499999999997 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076445+0000", 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003080396 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077482+0000", 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001036709 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077535+0000", 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3211000000000003e-05 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077592+0000", 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7664000000000002e-05 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8371 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.374494146000004, 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021386721000000001, 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8371 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056381+0000", 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056385+0000", 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056391+0000", 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6959999999999999e-06 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056396+0000", 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2549999999999999e-06 2024-09-16T13:17:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074355+0000", 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017959098999999999 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074503+0000", 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014797200000000001 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074955+0000", 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045213300000000002 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077607+0000", 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026519500000000001 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077660+0000", 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077725+0000", 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4991000000000002e-05 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077767+0000", 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8372 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.374473125000002, 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021557531000000001, 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056402+0000", 2024-09-16T13:17:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1399999999999997e-07 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056405+0000", 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6520000000000002e-06 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056411+0000", 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2009999999999996e-06 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056415+0000", 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.211e-06 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075482+0000", 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019067163000000002 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075546+0000", 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3516000000000006e-05 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075844+0000", 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029810999999999998 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077784+0000", 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019399350000000001 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077823+0000", 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9524000000000003e-05 2024-09-16T13:17:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077917+0000", 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4090000000000002e-05 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077959+0000", 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1615000000000001e-05 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8373 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.374452781999999, 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022897421000000001, 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3200000000000005e-07 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056427+0000", 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-06 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056433+0000", 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056437+0000", 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:17:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076470+0000", 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020032239 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076524+0000", 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4475000000000003e-05 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076996+0000", 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000471905 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.078749+0000", 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001752539 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079217+0000", 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046819400000000001 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079276+0000", 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9407000000000002e-05 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079319+0000", 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8398 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.345088990000001, 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017439282, 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8398 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:17:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085788+0000", 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4719999999999998e-06 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085793+0000", 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6179999999999997e-06 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085796+0000", 2024-09-16T13:17:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6680000000000001e-06 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099514+0000", 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013717895000000001 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099873+0000", 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035970799999999998 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100269+0000", 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000395382 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103068+0000", 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027992680000000002 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103114+0000", 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6189000000000001e-05 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103178+0000", 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3845999999999998e-05 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103225+0000", 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6888999999999998e-05 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8399 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.345047512999997, 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017551573000000001, 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8399 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085828+0000", 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085830+0000", 2024-09-16T13:17:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8609999999999998e-06 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085835+0000", 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.437e-06 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085837+0000", 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.571e-06 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100950+0000", 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015112537000000001 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101047+0000", 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6945000000000001e-05 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101468+0000", 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000420794 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103315+0000", 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018474769999999999 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103321+0000", 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103356+0000", 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4764000000000001e-05 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103379+0000", 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3102999999999999e-05 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8400 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.345034611999999, 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017960752, 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8400 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.53e-07 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085842+0000", 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2620000000000001e-06 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085847+0000", 2024-09-16T13:17:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2119999999999997e-06 2024-09-16T13:17:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085849+0000", 2024-09-16T13:17:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.204e-06 2024-09-16T13:17:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101995+0000", 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016146384999999999 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102100+0000", 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010482199999999999 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102466+0000", 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000365731 2024-09-16T13:17:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103389+0000", 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092367199999999995 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103702+0000", 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031220400000000001 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103754+0000", 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2318999999999998e-05 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103801+0000", 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6687999999999998e-05 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8539 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.186903057000002, 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018486425000000001, 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8539 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0500000000000003e-07 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243975+0000", 2024-09-16T13:17:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800000000000002e-06 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243982+0000", 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243986+0000", 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4100000000000001e-06 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258523+0000", 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014536604999999999 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258678+0000", 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155341 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.259218+0000", 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053930600000000003 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262325+0000", 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031076799999999998 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262367+0000", 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1480000000000003e-05 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262417+0000", 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0555e-05 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262458+0000", 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0556999999999998e-05 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8540 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.186883635999997, 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018906683000000001, 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8540 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243992+0000", 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.75e-07 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243995+0000", 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1219999999999999e-06 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244001+0000", 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5010000000000001e-06 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244006+0000", 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999996e-06 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260137+0000", 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016131 2024-09-16T13:17:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260200+0000", 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3782999999999996e-05 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260598+0000", 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039746600000000002 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262331+0000", 2024-09-16T13:17:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017327849999999999 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262786+0000", 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045546300000000003 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262851+0000", 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-05 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262898+0000", 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7098000000000002e-05 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8541 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.186864413000002, 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019366889000000002, 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8541 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244011+0000", 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244014+0000", 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.79e-06 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244020+0000", 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6370000000000002e-06 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244025+0000", 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3359999999999997e-06 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261122+0000", 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017097161 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261228+0000", 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105989 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261475+0000", 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024757000000000002 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263022+0000", 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015466760000000001 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263300+0000", 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027828100000000001 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263344+0000", 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4008999999999998e-05 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263377+0000", 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2832e-05 2024-09-16T13:17:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11776 3.1a 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 33.913461406000003, 2024-09-16T13:17:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027432719000000001, 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11776 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517415+0000", 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5239999999999999e-06 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526328+0000", 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089138059999999998 2024-09-16T13:17:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526362+0000", 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3775000000000001e-05 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526379+0000", 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6835000000000001e-05 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526467+0000", 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7703999999999999e-05 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526653+0000", 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186379 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.527624+0000", 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097080400000000002 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543313+0000", 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015689456000000001 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544696+0000", 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013822979999999999 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544784+0000", 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8419999999999997e-05 2024-09-16T13:17:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544846+0000", 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1717999999999994e-05 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:11778 3.1 e40/23 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head v 40'57, mlcod=40'55)", 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 33.906977378999997, 2024-09-16T13:17:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01745787, 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523898+0000", 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-06 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533067+0000", 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091691120000000001 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533101+0000", 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3733999999999999e-05 2024-09-16T13:17:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533115+0000", 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3410999999999999e-05 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533194+0000", 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9827000000000007e-05 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533223+0000", 2024-09-16T13:17:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8937000000000001e-05 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541297+0000", 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080739850000000005 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541355+0000", 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7708999999999999e-05 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11777 3.1a 3:59cb7a3f:::rbd_object_map.112d21cb8818:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 33.904472468000002, 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018649718999999999, 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11777 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526404+0000", 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.654e-06 2024-09-16T13:17:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526409+0000", 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.882e-06 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526424+0000", 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5328999999999998e-05 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526428+0000", 2024-09-16T13:17:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6219999999999998e-06 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542209+0000", 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015781765999999999 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542334+0000", 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125148 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542804+0000", 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046944899999999998 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543732+0000", 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092824699999999999 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544929+0000", 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001196522 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545004+0000", 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4962999999999994e-05 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545052+0000", 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8136999999999997e-05 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11779 3.9 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 33.883950886000001, 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020806181999999999, 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11779 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546926+0000", 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9769999999999999e-06 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552330+0000", 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054043980000000004 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552353+0000", 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2875999999999999e-05 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552362+0000", 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7299999999999994e-06 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552447+0000", 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5620999999999999e-05 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552710+0000", 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026273300000000002 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.553446+0000", 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073563299999999997 2024-09-16T13:17:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.559002+0000", 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055564580000000002 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567615+0000", 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086126680000000004 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567689+0000", 2024-09-16T13:17:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3839999999999998e-05 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567730+0000", 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1248000000000002e-05 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13736 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.132879709000001, 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018229967999999999, 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13736 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2e-07 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297998+0000", 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.824e-06 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298004+0000", 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8730000000000004e-06 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298008+0000", 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312651+0000", 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014643351000000001 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312740+0000", 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8466000000000003e-05 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313081+0000", 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034067899999999998 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315982+0000", 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002900943 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316051+0000", 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9029999999999995e-05 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316101+0000", 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0680000000000003e-05 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316225+0000", 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012359099999999999 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13737 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.132862588999998, 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018381208999999999, 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13737 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298015+0000", 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6869999999999999e-06 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298021+0000", 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7799999999999997e-06 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298027+0000", 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.071e-06 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313829+0000", 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015801696 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314019+0000", 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018995700000000001 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314379+0000", 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036036500000000002 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315988+0000", 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016084949999999999 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316290+0000", 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000302732 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316324+0000", 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3937999999999999e-05 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316393+0000", 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8966000000000003e-05 2024-09-16T13:17:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13738 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.132843607000002, 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021400311000000002, 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13738 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298032+0000", 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7400000000000003e-07 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298034+0000", 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5770000000000001e-06 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298040+0000", 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7010000000000001e-06 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298044+0000", 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2069999999999997e-06 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315033+0000", 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016988970999999999 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315106+0000", 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2723000000000002e-05 2024-09-16T13:17:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315428+0000", 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032191099999999997 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319249+0000", 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003821166 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319313+0000", 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319347+0000", 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4591999999999997e-05 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319431+0000", 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3893000000000005e-05 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13739 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.132826094999999, 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021506517999999999, 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13739 2024-09-16T13:17:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298049+0000", 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298052+0000", 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.819e-06 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298058+0000", 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6960000000000002e-06 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298061+0000", 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7560000000000001e-06 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316465+0000", 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018403744 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316747+0000", 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028175200000000002 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.317694+0000", 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000947115 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319304+0000", 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016102029999999999 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319446+0000", 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142008 2024-09-16T13:17:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319490+0000", 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3690999999999999e-05 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319555+0000", 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5191000000000007e-05 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13740 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.132772612, 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023881870999999999, 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13740 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298103+0000", 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298106+0000", 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1559999999999999e-06 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298120+0000", 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4474999999999999e-05 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298124+0000", 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1699999999999999e-06 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319635+0000", 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021510129999999999 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319762+0000", 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012746400000000001 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.320225+0000", 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046264800000000002 2024-09-16T13:17:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321629+0000", 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00140383 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321682+0000", 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3720000000000001e-05 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321757+0000", 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:17:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321984+0000", 2024-09-16T13:17:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022684199999999999 2024-09-16T13:17:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.555 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:51.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:51.557 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:51.552+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:51.658 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:51.652+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:51.666 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:51.666 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:51.666 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:51.667 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:51.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:51.759 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:51.752+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:51.781 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:51.781 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:51.781 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:51.781 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:51.781 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:51.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:51.859 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:51.851+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:51.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:51.893 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:51.893 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:51.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:51.893 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:51.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.893 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2523 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.893 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:51.893 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862657497999997, 2024-09-16T13:17:51.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014015147, 2024-09-16T13:17:51.893 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2523 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024656+0000", 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.131e-06 2024-09-16T13:17:51.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024660+0000", 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6990000000000001e-06 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024663+0000", 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0290000000000001e-06 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038456+0000", 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013793481 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038549+0000", 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2608000000000005e-05 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038669+0000", 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011982800000000001 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.897 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2524 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862643923999997, 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014381392, 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2524 2024-09-16T13:17:51.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024668+0000", 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8400000000000003e-07 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024670+0000", 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.849e-06 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024675+0000", 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.451e-06 2024-09-16T13:17:51.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024678+0000", 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8739999999999999e-06 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038723+0000", 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014044562999999999 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038811+0000", 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039049+0000", 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023761100000000001 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2525 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 557056~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:51.902 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862609577000001, 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014602618, 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2525 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024705+0000", 2024-09-16T13:17:51.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4650000000000001e-06 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024711+0000", 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024716+0000", 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1710000000000004e-06 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039096+0000", 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014380184000000001 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039187+0000", 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1065999999999999e-05 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039304+0000", 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117333 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2526 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1507328~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862591408, 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014823846, 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2526 2024-09-16T13:17:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:17:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024722+0000", 2024-09-16T13:17:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:17:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024729+0000", 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024733+0000", 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039337+0000", 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014604026000000001 2024-09-16T13:17:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039427+0000", 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0437000000000002e-05 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039544+0000", 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116234 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2527 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1605632~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862573644999998, 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015041116, 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2527 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024738+0000", 2024-09-16T13:17:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024746+0000", 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2130000000000008e-06 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024753+0000", 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024758+0000", 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7670000000000003e-06 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014820533 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039670+0000", 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1825999999999998e-05 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039779+0000", 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108424 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2528 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862548207000003, 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015245242000000001, 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2528 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024765+0000", 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1040000000000001e-06 2024-09-16T13:17:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024772+0000", 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024776+0000", 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039810+0000", 2024-09-16T13:17:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015034194000000001 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039897+0000", 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6864e-05 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040008+0000", 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110802 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2529 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2555904~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862529088000002, 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015471574, 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2529 2024-09-16T13:17:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024785+0000", 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5969999999999999e-06 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024800+0000", 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4681999999999999e-05 2024-09-16T13:17:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024806+0000", 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040039+0000", 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015233007999999999 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040143+0000", 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010322799999999999 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040254+0000", 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011092899999999999 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2530 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862498807999998, 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015667606000000001, 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2530 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024813+0000", 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024815+0000", 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5160000000000001e-06 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024827+0000", 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1915e-05 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024834+0000", 2024-09-16T13:17:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5180000000000001e-06 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040286+0000", 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015452565999999999 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040375+0000", 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040480+0000", 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105388 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2531 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3538944~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862471593000002, 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015867612999999999, 2024-09-16T13:17:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2531 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024843+0000", 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6960000000000001e-06 2024-09-16T13:17:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024856+0000", 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-05 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024863+0000", 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6069999999999996e-06 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040513+0000", 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650662999999999 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040599+0000", 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5439999999999995e-05 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040707+0000", 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108461 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2532 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862441087999997, 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061912000000001, 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2532 2024-09-16T13:17:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024871+0000", 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024874+0000", 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024881+0000", 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0570000000000004e-06 2024-09-16T13:17:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024886+0000", 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040742+0000", 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015856808 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040824+0000", 2024-09-16T13:17:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040932+0000", 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108378 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2533 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862420563000001, 2024-09-16T13:17:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016272282999999998, 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2533 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024893+0000", 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419999999999999e-06 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024900+0000", 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024903+0000", 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040965+0000", 2024-09-16T13:17:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061565999999999 2024-09-16T13:17:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041052+0000", 2024-09-16T13:17:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7213000000000007e-05 2024-09-16T13:17:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041163+0000", 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011066899999999999 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2534 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3604480~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862402031999999, 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016482746999999999, 2024-09-16T13:17:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2534 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024910+0000", 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1299999999999999e-07 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024912+0000", 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-09-16T13:17:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024920+0000", 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024924+0000", 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.036e-06 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041197+0000", 2024-09-16T13:17:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016273242 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041282+0000", 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5013999999999996e-05 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041392+0000", 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109567 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2535 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862381370000001, 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016691902000000002, 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2535 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7599999999999997e-07 2024-09-16T13:17:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024932+0000", 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024949+0000", 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7053000000000001e-05 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024961+0000", 2024-09-16T13:17:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041425+0000", 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016463851000000002 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041515+0000", 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9549999999999995e-05 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041622+0000", 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107132 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2536 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1638400~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862343564, 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016880869999999999, 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2536 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024971+0000", 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.615e-06 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024988+0000", 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6912999999999999e-05 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024993+0000", 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041653+0000", 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016660245000000001 2024-09-16T13:17:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041743+0000", 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0099999999999995e-05 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041848+0000", 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105404 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2537 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1671168~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862312811000002, 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017075346000000002, 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2537 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024999+0000", 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:17:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025002+0000", 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025009+0000", 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025013+0000", 2024-09-16T13:17:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1840000000000001e-06 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041886+0000", 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016873249999999999 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041971+0000", 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4487000000000006e-05 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042074+0000", 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102847 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2538 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2621440~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.86229264, 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017314057000000001, 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2538 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025019+0000", 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025021+0000", 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025033+0000", 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1985e-05 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025043+0000", 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3810000000000005e-06 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042107+0000", 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017064471000000001 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042203+0000", 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5810999999999995e-05 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042333+0000", 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012956099999999999 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2539 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.862264912000001, 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017500180000000001, 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2539 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025047+0000", 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025053+0000", 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025058+0000", 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025061+0000", 2024-09-16T13:17:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.072e-06 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042365+0000", 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017304885999999998 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042458+0000", 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2991000000000005e-05 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042546+0000", 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7904000000000004e-05 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:22787 3.17 3:edd0f01e:::rbd_object_map.112db5030c67:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.56988033, 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015269972, 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 22787 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317432+0000", 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.127e-06 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317439+0000", 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0500000000000003e-06 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317458+0000", 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9326000000000001e-05 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317467+0000", 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0669999999999998e-06 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329243+0000", 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011775551 2024-09-16T13:17:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329293+0000", 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9709000000000001e-05 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329505+0000", 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021237200000000001 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.330969+0000", 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014635290000000001 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332590+0000", 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001621485 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332655+0000", 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4670000000000003e-05 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332701+0000", 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6085999999999999e-05 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:23429 3.5 3:a41c7f85:::rbd_object_map.112dc49cabe2:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.7981841630000002, 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014415534000000001, 2024-09-16T13:17:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 23429 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089128+0000", 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7599999999999996e-07 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089140+0000", 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1939999999999999e-05 2024-09-16T13:17:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089164+0000", 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4430000000000002e-05 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089173+0000", 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8200000000000003e-06 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.097878+0000", 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087055320000000002 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.098035+0000", 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156537 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.098605+0000", 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057031900000000001 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103250+0000", 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046448119999999999 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103351+0000", 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100763 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103472+0000", 2024-09-16T13:17:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121091 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103542+0000", 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0513999999999996e-05 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:23430 3.5 3:a41c7f85:::rbd_object_map.112dc49cabe2:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.7981282729999997, 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016212550999999999, 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 23430 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089184+0000", 2024-09-16T13:17:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089192+0000", 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8730000000000007e-06 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089210+0000", 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8127000000000001e-05 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089217+0000", 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7130000000000005e-06 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.099775+0000", 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010557957999999999 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.099894+0000", 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011883999999999999 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.100458+0000", 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056368200000000003 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103330+0000", 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028723690000000001 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105122+0000", 2024-09-16T13:17:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017912430000000001 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105297+0000", 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175055 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105395+0000", 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8758000000000006e-05 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:51.990 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:51.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:51.992 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:51.984+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:52.018 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:52.043 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:52.061 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:52.081 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:52.091 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:52.083+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:52.102 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:52.103 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:52.103 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:52.103 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:52.103 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:52.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:52.123 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:52.144 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:52.165 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:52.167 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:52.168 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:52.188 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone21 from 0xda9ae90 (overlap 0x111c11) to 0x0 2024-09-16T13:17:52.192 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:52.184+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:52.219 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:52.219 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:52.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:52.293 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:52.284+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:52.325 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:52.325 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:52.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:52.383 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #20, image image_client.0-clone20 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent21 2024-09-16T13:17:52.394 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:52.384+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:52.397 INFO:teuthology.orchestra.run.smithi098.stdout:3710 write 0x7fc5743 thru 0x7fce867 (0x9125 bytes) 2024-09-16T13:17:52.416 INFO:teuthology.orchestra.run.smithi098.stdout:3711 punch from 0x1e3b4c1 to 0x1e4b223, (0xfd62 bytes) 2024-09-16T13:17:52.416 INFO:teuthology.orchestra.run.smithi098.stdout:3712 write 0x874768a thru 0x875495f (0xd2d6 bytes) 2024-09-16T13:17:52.433 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:52.433 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:52.440 INFO:teuthology.orchestra.run.smithi098.stdout:3713 writesame 0xde8e653 thru 0xde97a0d data_size 0x1(0x93bb bytes) 2024-09-16T13:17:52.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:52.449 INFO:teuthology.orchestra.run.smithi098.stdout:3714 read 0xda48060 thru 0xda51ac2 (0x9a63 bytes) 2024-09-16T13:17:52.449 INFO:teuthology.orchestra.run.smithi098.stdout:3716 read 0x391f060 thru 0x392d87d (0xe81e bytes) 2024-09-16T13:17:52.450 INFO:teuthology.orchestra.run.smithi098.stdout:3719 read 0x9b08c2e thru 0x9b16d90 (0xe163 bytes) 2024-09-16T13:17:52.450 INFO:teuthology.orchestra.run.smithi098.stdout:3720 read 0xc1ed410 thru 0xc1fa04d (0xcc3e bytes) 2024-09-16T13:17:52.450 INFO:teuthology.orchestra.run.smithi098.stdout:3721 write 0x74707a2 thru 0x74762bc (0x5b1b bytes) 2024-09-16T13:17:52.461 INFO:teuthology.orchestra.run.smithi098.stdout:3722 write 0x89167fe thru 0x89225a0 (0xbda3 bytes) 2024-09-16T13:17:52.470 INFO:teuthology.orchestra.run.smithi098.stdout:3723 write 0x68690f9 thru 0x686c441 (0x3349 bytes) 2024-09-16T13:17:52.473 INFO:teuthology.orchestra.run.smithi098.stdout:3724 punch from 0xbe7a72c to 0xbe881bb, (0xda8f bytes) 2024-09-16T13:17:52.481 INFO:teuthology.orchestra.run.smithi098.stdout:3725 trunc from 0xde97a0e to 0xa5b2596 2024-09-16T13:17:52.495 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:52.488+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:52.506 INFO:teuthology.orchestra.run.smithi098.stdout:3726 write 0x431491c thru 0x431b6d6 (0x6dbb bytes) 2024-09-16T13:17:52.523 INFO:teuthology.orchestra.run.smithi098.stdout:3730 write 0xa63ad4f thru 0xa642f65 (0x8217 bytes) 2024-09-16T13:17:52.526 INFO:teuthology.orchestra.run.smithi098.stdout:3731 punch from 0x9678a9e to 0x967df1b, (0x547d bytes) 2024-09-16T13:17:52.526 INFO:teuthology.orchestra.run.smithi098.stdout:3732 punch from 0x2f9fe2c to 0x2faef26, (0xf0fa bytes) 2024-09-16T13:17:52.526 INFO:teuthology.orchestra.run.smithi098.stdout:3734 read 0x1bb4115 thru 0x1bbd3b1 (0x929d bytes) 2024-09-16T13:17:52.526 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:52.527 INFO:teuthology.orchestra.run.smithi098.stdout:3736 writesame 0x30322ed thru 0x30360bf data_size 0x31(0x3dd3 bytes) 2024-09-16T13:17:52.536 INFO:teuthology.orchestra.run.smithi098.stdout:3737 read 0x4806515 thru 0x480ff63 (0x9a4f bytes) 2024-09-16T13:17:52.537 INFO:teuthology.orchestra.run.smithi098.stdout:3738 writesame 0x2be6174 thru 0x2be7b2a data_size 0x1(0x19b7 bytes) 2024-09-16T13:17:52.545 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:52.545 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:52.546 INFO:teuthology.orchestra.run.smithi098.stdout:3739 writesame 0x1e5239f thru 0x1e56753 data_size 0x1(0x43b5 bytes) 2024-09-16T13:17:52.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:52.566 INFO:teuthology.orchestra.run.smithi098.stdout:3740 read 0x8de6916 thru 0x8de730b (0x9f6 bytes) 2024-09-16T13:17:52.567 INFO:teuthology.orchestra.run.smithi098.stdout:3741 punch from 0x37dd898 to 0x37ec7a9, (0xef11 bytes) 2024-09-16T13:17:52.567 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:52.586 INFO:teuthology.orchestra.run.smithi098.stdout:3743 trunc from 0xa642f66 to 0xcb2045 2024-09-16T13:17:52.596 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:52.588+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:52.649 INFO:teuthology.orchestra.run.smithi098.stdout:3744 write 0xad737a9 thru 0xad76942 (0x319a bytes) 2024-09-16T13:17:52.656 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:52.656 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:52.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:52.669 INFO:teuthology.orchestra.run.smithi098.stdout:3746 writesame 0xcab19b3 thru 0xcab6ba3 data_size 0x1(0x51f1 bytes) 2024-09-16T13:17:52.669 INFO:teuthology.orchestra.run.smithi098.stdout:3747 punch from 0x630d1d4 to 0x6313327, (0x6153 bytes) 2024-09-16T13:17:52.669 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:52.669 INFO:teuthology.orchestra.run.smithi098.stdout:3749 punch from 0x865e04d to 0x86647f0, (0x67a3 bytes) 2024-09-16T13:17:52.669 INFO:teuthology.orchestra.run.smithi098.stdout:3750 write 0x4555ad5 thru 0x4561c7f (0xc1ab bytes) 2024-09-16T13:17:52.673 INFO:teuthology.orchestra.run.smithi098.stdout:3751 punch from 0x51f594e to 0x5204288, (0xe93a bytes) 2024-09-16T13:17:52.673 INFO:teuthology.orchestra.run.smithi098.stdout:3753 read 0x55cf75a thru 0x55d85ad (0x8e54 bytes) 2024-09-16T13:17:52.674 INFO:teuthology.orchestra.run.smithi098.stdout:3754 punch from 0x3a22ef7 to 0x3a2dbb0, (0xacb9 bytes) 2024-09-16T13:17:52.674 INFO:teuthology.orchestra.run.smithi098.stdout:3755 writesame 0x61caaca thru 0x61cae23 data_size 0xb(0x35a bytes) 2024-09-16T13:17:52.676 INFO:teuthology.orchestra.run.smithi098.stdout:3756 write 0xb6aff9b thru 0xb6b8cd6 (0x8d3c bytes) 2024-09-16T13:17:52.683 INFO:teuthology.orchestra.run.smithi098.stdout:3757 read 0xb940c55 thru 0xb94aedf (0xa28b bytes) 2024-09-16T13:17:52.688 INFO:teuthology.orchestra.run.smithi098.stdout:3759 trunc from 0xcab6ba4 to 0xee1b3af 2024-09-16T13:17:52.697 INFO:teuthology.orchestra.run.smithi098.stdout:3760 write 0xe4ebc35 thru 0xe4ec029 (0x3f5 bytes) 2024-09-16T13:17:52.698 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:52.698 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:52.698 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-16T13:17:52.698 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:17:52.698 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:17:52.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-16T13:17:52.700 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:52.688+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:52.702 INFO:teuthology.orchestra.run.smithi098.stdout:3762 clone 23 order 21 su 1048576 sc 11 2024-09-16T13:17:52.788 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:52.789 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:52.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:52.799 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:52.788+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:52.803 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:52.803 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T13:17:52.803 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:52.899 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:52.895+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:52.901 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:52.901 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:52.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:53.000 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:52.992+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:53.004 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:53.004 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:53.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:53.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:17:53.100 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:53.091+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:53.110 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:53.110 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:53.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:53.201 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:53.192+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:53.211 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone22 from 0xee1b3af (overlap 0xcb2045) to 0x659022 2024-09-16T13:17:53.217 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:53.217 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:53.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:53.303 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:53.295+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:53.339 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:53.339 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:53.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:53.381 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #21, image image_client.0-clone21 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent22 2024-09-16T13:17:53.384 INFO:teuthology.orchestra.run.smithi098.stdout:3765 write 0x4c130f2 thru 0x4c1efbf (0xbece bytes) 2024-09-16T13:17:53.399 INFO:teuthology.orchestra.run.smithi098.stdout:3766 read 0x58b5770 thru 0x58bbc5f (0x64f0 bytes) 2024-09-16T13:17:53.400 INFO:teuthology.orchestra.run.smithi098.stdout:3767 writesame 0x4d615a1 thru 0x4d6cc72 data_size 0x7(0xb6d2 bytes) 2024-09-16T13:17:53.403 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:53.396+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:53.414 INFO:teuthology.orchestra.run.smithi098.stdout:3768 writesame 0xb5ed53c thru 0xb5f45fb data_size 0x58(0x70c0 bytes) 2024-09-16T13:17:53.430 INFO:teuthology.orchestra.run.smithi098.stdout:3769 trunc from 0xee1b3af to 0xb7f61d2 2024-09-16T13:17:53.453 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:53.453 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:53.455 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.455 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.455 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.455 INFO:teuthology.orchestra.run.smithi098.stdout:3775 punch from 0x1119752 to 0x112547b, (0xbd29 bytes) 2024-09-16T13:17:53.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:53.467 INFO:teuthology.orchestra.run.smithi098.stdout:3777 write 0xd66cfcb thru 0xd678a02 (0xba38 bytes) 2024-09-16T13:17:53.474 INFO:teuthology.orchestra.run.smithi098.stdout:3778 read 0x2a6ef40 thru 0x2a6ff0d (0xfce bytes) 2024-09-16T13:17:53.475 INFO:teuthology.orchestra.run.smithi098.stdout:3779 writesame 0x38e329c thru 0x38ee25c data_size 0x1(0xafc1 bytes) 2024-09-16T13:17:53.504 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:53.496+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:53.511 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.534 INFO:teuthology.orchestra.run.smithi098.stdout:3784 trunc from 0xd678a03 to 0x13fd1c 2024-09-16T13:17:53.570 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.570 INFO:teuthology.orchestra.run.smithi098.stdout:3786 read 0x111dc8 thru 0x118dad (0x6fe6 bytes) 2024-09-16T13:17:53.577 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:53.577 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:53.588 INFO:teuthology.orchestra.run.smithi098.stdout:3787 trunc from 0x13fd1c to 0xadd147b 2024-09-16T13:17:53.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:53.601 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.601 INFO:teuthology.orchestra.run.smithi098.stdout:3790 punch from 0x56bfad6 to 0x56cb02e, (0xb558 bytes) 2024-09-16T13:17:53.601 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.601 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.601 INFO:teuthology.orchestra.run.smithi098.stdout:3794 read 0x2bbf7f9 thru 0x2bce7f8 (0xf000 bytes) 2024-09-16T13:17:53.601 INFO:teuthology.orchestra.run.smithi098.stdout:3795 punch from 0xa24a9c3 to 0xa24f808, (0x4e45 bytes) 2024-09-16T13:17:53.601 INFO:teuthology.orchestra.run.smithi098.stdout:3796 read 0x5f40f7a thru 0x5f43174 (0x21fb bytes) 2024-09-16T13:17:53.620 INFO:teuthology.orchestra.run.smithi098.stdout:3797 write 0xeb2225e thru 0xeb262f6 (0x4099 bytes) 2024-09-16T13:17:53.620 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:53.612+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:53.627 INFO:teuthology.orchestra.run.smithi098.stdout:3798 punch from 0x6faabda to 0x6fb5a88, (0xaeae bytes) 2024-09-16T13:17:53.627 INFO:teuthology.orchestra.run.smithi098.stdout:3799 punch from 0x854dbf0 to 0x8550b29, (0x2f39 bytes) 2024-09-16T13:17:53.631 INFO:teuthology.orchestra.run.smithi098.stdout:3800 trunc from 0xeb262f7 to 0xcac500f 2024-09-16T13:17:53.652 INFO:teuthology.orchestra.run.smithi098.stdout:3801 write 0x5b89205 thru 0x5b950c3 (0xbebf bytes) 2024-09-16T13:17:53.659 INFO:teuthology.orchestra.run.smithi098.stdout:3802 read 0x8ea08a6 thru 0x8ea16da (0xe35 bytes) 2024-09-16T13:17:53.659 INFO:teuthology.orchestra.run.smithi098.stdout:3803 writesame 0x97ef8ef thru 0x97f8a9e data_size 0x94(0x91b0 bytes) 2024-09-16T13:17:53.667 INFO:teuthology.orchestra.run.smithi098.stdout:3805 write 0xad5c0f1 thru 0xad61283 (0x5193 bytes) 2024-09-16T13:17:53.673 INFO:teuthology.orchestra.run.smithi098.stdout:3808 writesame 0x5b42498 thru 0x5b496ea data_size 0x25(0x7253 bytes) 2024-09-16T13:17:53.677 INFO:teuthology.orchestra.run.smithi098.stdout:3809 write 0x98b4b5d thru 0x98c11ce (0xc672 bytes) 2024-09-16T13:17:53.684 INFO:teuthology.orchestra.run.smithi098.stdout:3810 write 0xc715b9b thru 0xc716f7a (0x13e0 bytes) 2024-09-16T13:17:53.690 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.690 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.690 INFO:teuthology.orchestra.run.smithi098.stdout:3813 punch from 0x33d1531 to 0x33db765, (0xa234 bytes) 2024-09-16T13:17:53.690 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.690 INFO:teuthology.orchestra.run.smithi098.stdout:3816 write 0x9194d36 thru 0x9195434 (0x6ff bytes) 2024-09-16T13:17:53.703 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:53.703 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:53.705 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:53.696+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:53.709 INFO:teuthology.orchestra.run.smithi098.stdout:3820 write 0xd51e73c thru 0xd52ad39 (0xc5fe bytes) 2024-09-16T13:17:53.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:53.717 INFO:teuthology.orchestra.run.smithi098.stdout:3822 write 0x997990c thru 0x9988a2c (0xf121 bytes) 2024-09-16T13:17:53.725 INFO:teuthology.orchestra.run.smithi098.stdout:3823 write 0xb71c69 thru 0xb734cd (0x1865 bytes) 2024-09-16T13:17:53.728 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.728 INFO:teuthology.orchestra.run.smithi098.stdout:3825 read 0xbd2048d thru 0xbd2fe4a (0xf9be bytes) 2024-09-16T13:17:53.728 INFO:teuthology.orchestra.run.smithi098.stdout:3828 write 0x6527a68 thru 0x6527d91 (0x32a bytes) 2024-09-16T13:17:53.744 INFO:teuthology.orchestra.run.smithi098.stdout:3830 trunc from 0xd52ad3a to 0x9a1db32 2024-09-16T13:17:53.781 INFO:teuthology.orchestra.run.smithi098.stdout:3831 trunc from 0x9a1db32 to 0x6920f56 2024-09-16T13:17:53.806 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:53.799+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:53.806 INFO:teuthology.orchestra.run.smithi098.stdout:3832 trunc from 0x6920f56 to 0x4f85636 2024-09-16T13:17:53.818 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:53.818 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:53.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:53.828 INFO:teuthology.orchestra.run.smithi098.stdout:3833 write 0x3459cf9 thru 0x3468f22 (0xf22a bytes) 2024-09-16T13:17:53.865 INFO:teuthology.orchestra.run.smithi098.stdout:3834 write 0xd0431f9 thru 0xd0502d7 (0xd0df bytes) 2024-09-16T13:17:53.873 INFO:teuthology.orchestra.run.smithi098.stdout:3835 write 0xaef4dbf thru 0xaef6679 (0x18bb bytes) 2024-09-16T13:17:53.881 INFO:teuthology.orchestra.run.smithi098.stdout:3836 write 0x2bd9ae thru 0x2ccf3c (0xf58f bytes) 2024-09-16T13:17:53.906 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:53.899+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:53.907 INFO:teuthology.orchestra.run.smithi098.stdout:3837 trunc from 0xd0502d8 to 0x2213fa2 2024-09-16T13:17:53.930 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:53.932 INFO:teuthology.orchestra.run.smithi098.stdout:3839 trunc from 0x2213fa2 to 0x10e9cce 2024-09-16T13:17:53.933 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:53.933 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:53.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:53.969 INFO:teuthology.orchestra.run.smithi098.stdout:3840 write 0x81fdb3d thru 0x820b6ea (0xdbae bytes) 2024-09-16T13:17:54.000 INFO:teuthology.orchestra.run.smithi098.stdout:3841 writesame 0xd38be2d thru 0xd38f100 data_size 0x2(0x32d4 bytes) 2024-09-16T13:17:54.000 INFO:teuthology.orchestra.run.smithi098.stdout:3842 writesame 0x10bd5b9 thru 0x10c2cfd data_size 0xb(0x5745 bytes) 2024-09-16T13:17:54.007 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:53.999+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:54.009 INFO:teuthology.orchestra.run.smithi098.stdout:3843 punch from 0x4778f14 to 0x477a22c, (0x1318 bytes) 2024-09-16T13:17:54.010 INFO:teuthology.orchestra.run.smithi098.stdout:3844 writesame 0x778f80b thru 0x7790b14 data_size 0x1(0x130a bytes) 2024-09-16T13:17:54.018 INFO:teuthology.orchestra.run.smithi098.stdout:3846 trunc from 0xd38f101 to 0xdbdc586 2024-09-16T13:17:54.033 INFO:teuthology.orchestra.run.smithi098.stdout:3847 read 0x7bfe431 thru 0x7c08b87 (0xa757 bytes) 2024-09-16T13:17:54.039 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:17:54.049 INFO:teuthology.orchestra.run.smithi098.stdout:3848 trunc from 0xdbdc586 to 0x4a06dc8 2024-09-16T13:17:54.074 INFO:teuthology.orchestra.run.smithi098.stdout:3849 punch from 0x47264d2 to 0x472bb6e, (0x569c bytes) 2024-09-16T13:17:54.079 INFO:teuthology.orchestra.run.smithi098.stdout:3850 trunc from 0x4a06dc8 to 0x7aad1a5 2024-09-16T13:17:54.085 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:54.085 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:54.091 INFO:teuthology.orchestra.run.smithi098.stdout:3851 read 0x79b2867 thru 0x79bce14 (0xa5ae bytes) 2024-09-16T13:17:54.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:54.114 INFO:teuthology.orchestra.run.smithi098.stdout:3852 write 0xc93c1cb thru 0xc93d4f3 (0x1329 bytes) 2024-09-16T13:17:54.130 INFO:teuthology.orchestra.run.smithi098.stdout:3853 trunc from 0xc93d4f4 to 0xa630a81 2024-09-16T13:17:54.141 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:54.135+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:54.159 INFO:teuthology.orchestra.run.smithi098.stdout:3855 trunc from 0xa630a81 to 0x6b5e2dd 2024-09-16T13:17:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:54.196 INFO:teuthology.orchestra.run.smithi098.stdout:3858 write 0xc160508 thru 0xc16b26b (0xad64 bytes) 2024-09-16T13:17:54.208 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:54.208 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:54.208 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:54.200+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:54.216 INFO:teuthology.orchestra.run.smithi098.stdout:3859 writesame 0xcdce999 thru 0xcdd2c27 data_size 0x1(0x428f bytes) 2024-09-16T13:17:54.216 INFO:teuthology.orchestra.run.smithi098.stdout:3860 punch from 0xb101dc1 to 0xb107d25, (0x5f64 bytes) 2024-09-16T13:17:54.217 INFO:teuthology.orchestra.run.smithi098.stdout:3861 read 0x18f69f0 thru 0x19020a1 (0xb6b2 bytes) 2024-09-16T13:17:54.217 INFO:teuthology.orchestra.run.smithi098.stdout:3864 punch from 0xbadf906 to 0xbaedd9f, (0xe499 bytes) 2024-09-16T13:17:54.217 INFO:teuthology.orchestra.run.smithi098.stdout:3865 read 0x3b069c1 thru 0x3b12a46 (0xc086 bytes) 2024-09-16T13:17:54.217 INFO:teuthology.orchestra.run.smithi098.stdout:3866 write 0x7fb785 thru 0x8045ad (0x8e29 bytes) 2024-09-16T13:17:54.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:54.224 INFO:teuthology.orchestra.run.smithi098.stdout:3867 read 0x7c1910d thru 0x7c26792 (0xd686 bytes) 2024-09-16T13:17:54.224 INFO:teuthology.orchestra.run.smithi098.stdout:3868 write 0x1ae94c7 thru 0x1af2f20 (0x9a5a bytes) 2024-09-16T13:17:54.230 INFO:teuthology.orchestra.run.smithi098.stdout:3869 read 0x5382e40 thru 0x5385187 (0x2348 bytes) 2024-09-16T13:17:54.231 INFO:teuthology.orchestra.run.smithi098.stdout:3870 writesame 0x70f0aa8 thru 0x70f0dd9 data_size 0x1(0x332 bytes) 2024-09-16T13:17:54.237 INFO:teuthology.orchestra.run.smithi098.stdout:3873 read 0x4413741 thru 0x4413c5c (0x51c bytes) 2024-09-16T13:17:54.237 INFO:teuthology.orchestra.run.smithi098.stdout:3874 punch from 0x265298d to 0x2656470, (0x3ae3 bytes) 2024-09-16T13:17:54.237 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.237 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.237 INFO:teuthology.orchestra.run.smithi098.stdout:3880 read 0x7c4d58a thru 0x7c50ebe (0x3935 bytes) 2024-09-16T13:17:54.237 INFO:teuthology.orchestra.run.smithi098.stdout:3881 punch from 0x506644e to 0x506909b, (0x2c4d bytes) 2024-09-16T13:17:54.237 INFO:teuthology.orchestra.run.smithi098.stdout:3882 write 0x17880e8 thru 0x178d667 (0x5580 bytes) 2024-09-16T13:17:54.244 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.244 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.249 INFO:teuthology.orchestra.run.smithi098.stdout:3888 trunc from 0xcdd2c28 to 0xa1273d1 2024-09-16T13:17:54.268 INFO:teuthology.orchestra.run.smithi098.stdout:3889 write 0x2910b89 thru 0x291f4ad (0xe925 bytes) 2024-09-16T13:17:54.279 INFO:teuthology.orchestra.run.smithi098.stdout:3891 trunc from 0xa1273d1 to 0xc884df1 2024-09-16T13:17:54.289 INFO:teuthology.orchestra.run.smithi098.stdout:3892 punch from 0xd150d1 to 0xd21493, (0xc3c2 bytes) 2024-09-16T13:17:54.290 INFO:teuthology.orchestra.run.smithi098.stdout:3894 trunc from 0xc884df1 to 0xd4942c2 2024-09-16T13:17:54.300 INFO:teuthology.orchestra.run.smithi098.stdout:3895 read 0x32acef2 thru 0x32b806a (0xb179 bytes) 2024-09-16T13:17:54.300 INFO:teuthology.orchestra.run.smithi098.stdout:3896 punch from 0x7e9ef1d to 0x7ea5c6a, (0x6d4d bytes) 2024-09-16T13:17:54.300 INFO:teuthology.orchestra.run.smithi098.stdout:3897 read 0x74cf3be thru 0x74d751c (0x815f bytes) 2024-09-16T13:17:54.306 INFO:teuthology.orchestra.run.smithi098.stdout:3899 trunc from 0xd4942c2 to 0xa5d53bd 2024-09-16T13:17:54.309 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:54.299+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:54.315 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.315 INFO:teuthology.orchestra.run.smithi098.stdout:3902 punch from 0x158987f to 0x1591c32, (0x83b3 bytes) 2024-09-16T13:17:54.315 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.315 INFO:teuthology.orchestra.run.smithi098.stdout:3906 read 0x7a28d75 thru 0x7a2c2a9 (0x3535 bytes) 2024-09-16T13:17:54.316 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.316 INFO:teuthology.orchestra.run.smithi098.stdout:3908 punch from 0x610bc1c to 0x6113fbf, (0x83a3 bytes) 2024-09-16T13:17:54.316 INFO:teuthology.orchestra.run.smithi098.stdout:3910 read 0x6a36f0e thru 0x6a36ff2 (0xe5 bytes) 2024-09-16T13:17:54.316 INFO:teuthology.orchestra.run.smithi098.stdout:3912 write 0x398cf63 thru 0x3992488 (0x5526 bytes) 2024-09-16T13:17:54.320 INFO:teuthology.orchestra.run.smithi098.stdout:3913 read 0x4cfb918 thru 0x4cfc4d9 (0xbc2 bytes) 2024-09-16T13:17:54.321 INFO:teuthology.orchestra.run.smithi098.stdout:3915 punch from 0x820ca32 to 0x8212110, (0x56de bytes) 2024-09-16T13:17:54.321 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:54.321 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:54.331 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:54.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:54.334 INFO:teuthology.orchestra.run.smithi098.stdout:3916 write 0xc916e13 thru 0xc921eb2 (0xb0a0 bytes) 2024-09-16T13:17:54.341 INFO:teuthology.orchestra.run.smithi098.stdout:3918 write 0xf57881 thru 0xf5f3cf (0x7b4f bytes) 2024-09-16T13:17:54.348 INFO:teuthology.orchestra.run.smithi098.stdout:3919 write 0x47ae196 thru 0x47ba854 (0xc6bf bytes) 2024-09-16T13:17:54.351 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:54.353 INFO:teuthology.orchestra.run.smithi098.stdout:3922 punch from 0x79fe488 to 0x7a00509, (0x2081 bytes) 2024-09-16T13:17:54.353 INFO:teuthology.orchestra.run.smithi098.stdout:3923 write 0x94b9556 thru 0x94c8b63 (0xf60e bytes) 2024-09-16T13:17:54.368 INFO:teuthology.orchestra.run.smithi098.stdout:3925 write 0xc9dba6f thru 0xc9e84e3 (0xca75 bytes) 2024-09-16T13:17:54.372 INFO:teuthology.orchestra.run.smithi098.stdout:3927 read 0x72a063 thru 0x7382fe (0xe29c bytes) 2024-09-16T13:17:54.372 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.373 INFO:teuthology.orchestra.run.smithi098.stdout:3929 writesame 0xb71692c thru 0xb724b60 data_size 0x61(0xe235 bytes) 2024-09-16T13:17:54.377 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:54.380 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.380 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.393 INFO:teuthology.orchestra.run.smithi098.stdout:3933 trunc from 0xc9e84e4 to 0x5903f82 2024-09-16T13:17:54.398 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:54.409 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:54.400+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:54.414 INFO:teuthology.orchestra.run.smithi098.stdout:3934 punch from 0x2d8ddba to 0x2d93063, (0x52a9 bytes) 2024-09-16T13:17:54.415 INFO:teuthology.orchestra.run.smithi098.stdout:3935 read 0xaf6ae0 thru 0xaf80f9 (0x161a bytes) 2024-09-16T13:17:54.415 INFO:teuthology.orchestra.run.smithi098.stdout:3936 punch from 0x2312deb to 0x231a5e8, (0x77fd bytes) 2024-09-16T13:17:54.415 INFO:teuthology.orchestra.run.smithi098.stdout:3937 punch from 0x32e4797 to 0x32e87ec, (0x4055 bytes) 2024-09-16T13:17:54.415 INFO:teuthology.orchestra.run.smithi098.stdout:3938 read 0x50028fe thru 0x500bdaa (0x94ad bytes) 2024-09-16T13:17:54.415 INFO:teuthology.orchestra.run.smithi098.stdout:3940 read 0x107a23f thru 0x1084ffc (0xadbe bytes) 2024-09-16T13:17:54.416 INFO:teuthology.orchestra.run.smithi098.stdout:3941 punch from 0x2d9472a to 0x2d9632a, (0x1c00 bytes) 2024-09-16T13:17:54.416 INFO:teuthology.orchestra.run.smithi098.stdout:3942 read 0x4d029ff thru 0x4d06e47 (0x4449 bytes) 2024-09-16T13:17:54.416 INFO:teuthology.orchestra.run.smithi098.stdout:3945 punch from 0x1c08138 to 0x1c123eb, (0xa2b3 bytes) 2024-09-16T13:17:54.421 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:54.422 INFO:teuthology.orchestra.run.smithi098.stdout:3946 trunc from 0x5903f82 to 0x19d45e9 2024-09-16T13:17:54.437 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8369 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.376372437999997, 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020058919000000001, 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8369 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056335+0000", 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056338+0000", 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056344+0000", 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8150000000000002e-06 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056348+0000", 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.070951+0000", 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014603055 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071060+0000", 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109462 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071508+0000", 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044824200000000001 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074304+0000", 2024-09-16T13:17:54.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027951719999999998 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076331+0000", 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020276529999999999 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076369+0000", 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7557e-05 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076393+0000", 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4445999999999999e-05 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8370 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.376355119999999, 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021240851000000002, 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8370 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056355+0000", 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.667e-06 2024-09-16T13:17:54.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056360+0000", 2024-09-16T13:17:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6620000000000002e-06 2024-09-16T13:17:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056366+0000", 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072672+0000", 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016306157000000002 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072746+0000", 2024-09-16T13:17:54.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3648999999999999e-05 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.073364+0000", 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061885499999999997 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076445+0000", 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003080396 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077482+0000", 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001036709 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077535+0000", 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3211000000000003e-05 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077592+0000", 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7664000000000002e-05 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.450 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8371 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.376326325000001, 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021386721000000001, 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.451 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8371 2024-09-16T13:17:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:17:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056381+0000", 2024-09-16T13:17:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-16T13:17:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056385+0000", 2024-09-16T13:17:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:17:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056391+0000", 2024-09-16T13:17:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6959999999999999e-06 2024-09-16T13:17:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056396+0000", 2024-09-16T13:17:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2549999999999999e-06 2024-09-16T13:17:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074355+0000", 2024-09-16T13:17:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017959098999999999 2024-09-16T13:17:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074503+0000", 2024-09-16T13:17:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014797200000000001 2024-09-16T13:17:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074955+0000", 2024-09-16T13:17:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045213300000000002 2024-09-16T13:17:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077607+0000", 2024-09-16T13:17:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026519500000000001 2024-09-16T13:17:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077660+0000", 2024-09-16T13:17:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:17:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077725+0000", 2024-09-16T13:17:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4991000000000002e-05 2024-09-16T13:17:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077767+0000", 2024-09-16T13:17:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:17:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.458 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8372 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.376305303999999, 2024-09-16T13:17:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021557531000000001, 2024-09-16T13:17:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.459 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-16T13:17:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:17:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056402+0000", 2024-09-16T13:17:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1399999999999997e-07 2024-09-16T13:17:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056405+0000", 2024-09-16T13:17:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6520000000000002e-06 2024-09-16T13:17:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056411+0000", 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2009999999999996e-06 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056415+0000", 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.211e-06 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075482+0000", 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019067163000000002 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075546+0000", 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3516000000000006e-05 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075844+0000", 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029810999999999998 2024-09-16T13:17:54.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077784+0000", 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019399350000000001 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077823+0000", 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9524000000000003e-05 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077917+0000", 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4090000000000002e-05 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077959+0000", 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1615000000000001e-05 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8373 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.466 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.376284961000003, 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022897421000000001, 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3200000000000005e-07 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056427+0000", 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-06 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056433+0000", 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056437+0000", 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076470+0000", 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020032239 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076524+0000", 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4475000000000003e-05 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076996+0000", 2024-09-16T13:17:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000471905 2024-09-16T13:17:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.078749+0000", 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001752539 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079217+0000", 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046819400000000001 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079276+0000", 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9407000000000002e-05 2024-09-16T13:17:54.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079319+0000", 2024-09-16T13:17:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:17:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.473 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8398 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.346921168999998, 2024-09-16T13:17:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017439282, 2024-09-16T13:17:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8398 2024-09-16T13:17:54.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:54.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:17:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:17:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085788+0000", 2024-09-16T13:17:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4719999999999998e-06 2024-09-16T13:17:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085793+0000", 2024-09-16T13:17:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6179999999999997e-06 2024-09-16T13:17:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085796+0000", 2024-09-16T13:17:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6680000000000001e-06 2024-09-16T13:17:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099514+0000", 2024-09-16T13:17:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013717895000000001 2024-09-16T13:17:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099873+0000", 2024-09-16T13:17:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035970799999999998 2024-09-16T13:17:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100269+0000", 2024-09-16T13:17:54.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000395382 2024-09-16T13:17:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103068+0000", 2024-09-16T13:17:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027992680000000002 2024-09-16T13:17:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103114+0000", 2024-09-16T13:17:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6189000000000001e-05 2024-09-16T13:17:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103178+0000", 2024-09-16T13:17:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3845999999999998e-05 2024-09-16T13:17:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103225+0000", 2024-09-16T13:17:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6888999999999998e-05 2024-09-16T13:17:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8399 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.346879692000002, 2024-09-16T13:17:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017551573000000001, 2024-09-16T13:17:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.483 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8399 2024-09-16T13:17:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:54.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:17:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085828+0000", 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085830+0000", 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8609999999999998e-06 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085835+0000", 2024-09-16T13:17:54.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.437e-06 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085837+0000", 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.571e-06 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100950+0000", 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015112537000000001 2024-09-16T13:17:54.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101047+0000", 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6945000000000001e-05 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101468+0000", 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000420794 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103315+0000", 2024-09-16T13:17:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018474769999999999 2024-09-16T13:17:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103321+0000", 2024-09-16T13:17:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103356+0000", 2024-09-16T13:17:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4764000000000001e-05 2024-09-16T13:17:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103379+0000", 2024-09-16T13:17:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3102999999999999e-05 2024-09-16T13:17:54.490 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8400 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.346866790999997, 2024-09-16T13:17:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017960752, 2024-09-16T13:17:54.491 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8400 2024-09-16T13:17:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:54.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:17:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.53e-07 2024-09-16T13:17:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085842+0000", 2024-09-16T13:17:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2620000000000001e-06 2024-09-16T13:17:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085847+0000", 2024-09-16T13:17:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2119999999999997e-06 2024-09-16T13:17:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085849+0000", 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.204e-06 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101995+0000", 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016146384999999999 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102100+0000", 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010482199999999999 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102466+0000", 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000365731 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103389+0000", 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092367199999999995 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103702+0000", 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031220400000000001 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103754+0000", 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2318999999999998e-05 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103801+0000", 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6687999999999998e-05 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8539 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.188735235999999, 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018486425000000001, 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.499 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8539 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0500000000000003e-07 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243975+0000", 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800000000000002e-06 2024-09-16T13:17:54.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243982+0000", 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243986+0000", 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4100000000000001e-06 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258523+0000", 2024-09-16T13:17:54.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014536604999999999 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258678+0000", 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155341 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.259218+0000", 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053930600000000003 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262325+0000", 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031076799999999998 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262367+0000", 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1480000000000003e-05 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262417+0000", 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0555e-05 2024-09-16T13:17:54.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262458+0000", 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0556999999999998e-05 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8540 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:54.505 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.188715815000002, 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018906683000000001, 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8540 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:54.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243992+0000", 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.75e-07 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243995+0000", 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1219999999999999e-06 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244001+0000", 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5010000000000001e-06 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244006+0000", 2024-09-16T13:17:54.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999996e-06 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260137+0000", 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016131 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260200+0000", 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3782999999999996e-05 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260598+0000", 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039746600000000002 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262331+0000", 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017327849999999999 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262786+0000", 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045546300000000003 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262851+0000", 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-05 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262898+0000", 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7098000000000002e-05 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8541 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:17:54.511 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.188696591999999, 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019366889000000002, 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8541 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244011+0000", 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244014+0000", 2024-09-16T13:17:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.79e-06 2024-09-16T13:17:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244020+0000", 2024-09-16T13:17:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6370000000000002e-06 2024-09-16T13:17:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244025+0000", 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3359999999999997e-06 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261122+0000", 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017097161 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261228+0000", 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105989 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261475+0000", 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024757000000000002 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263022+0000", 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015466760000000001 2024-09-16T13:17:54.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263300+0000", 2024-09-16T13:17:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027828100000000001 2024-09-16T13:17:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263344+0000", 2024-09-16T13:17:54.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4008999999999998e-05 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263377+0000", 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2832e-05 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.518 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11776 3.1a 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.915293585000001, 2024-09-16T13:17:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027432719000000001, 2024-09-16T13:17:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11776 2024-09-16T13:17:54.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517415+0000", 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5239999999999999e-06 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526328+0000", 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089138059999999998 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526362+0000", 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3775000000000001e-05 2024-09-16T13:17:54.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526379+0000", 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6835000000000001e-05 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526467+0000", 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7703999999999999e-05 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526653+0000", 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186379 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.527624+0000", 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097080400000000002 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543313+0000", 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015689456000000001 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544696+0000", 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013822979999999999 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544784+0000", 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8419999999999997e-05 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544846+0000", 2024-09-16T13:17:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1717999999999994e-05 2024-09-16T13:17:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:11778 3.1 e40/23 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head v 40'57, mlcod=40'55)", 2024-09-16T13:17:54.525 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.908809558000002, 2024-09-16T13:17:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01745787, 2024-09-16T13:17:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:17:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523898+0000", 2024-09-16T13:17:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-06 2024-09-16T13:17:54.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533067+0000", 2024-09-16T13:17:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091691120000000001 2024-09-16T13:17:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533101+0000", 2024-09-16T13:17:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3733999999999999e-05 2024-09-16T13:17:54.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533115+0000", 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3410999999999999e-05 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533194+0000", 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9827000000000007e-05 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533223+0000", 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8937000000000001e-05 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541297+0000", 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080739850000000005 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541355+0000", 2024-09-16T13:17:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7708999999999999e-05 2024-09-16T13:17:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11777 3.1a 3:59cb7a3f:::rbd_object_map.112d21cb8818:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:54.531 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.906304646999999, 2024-09-16T13:17:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018649718999999999, 2024-09-16T13:17:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11777 2024-09-16T13:17:54.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:17:54.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526404+0000", 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.654e-06 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526409+0000", 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.882e-06 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526424+0000", 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5328999999999998e-05 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526428+0000", 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6219999999999998e-06 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542209+0000", 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015781765999999999 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542334+0000", 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125148 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542804+0000", 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046944899999999998 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543732+0000", 2024-09-16T13:17:54.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092824699999999999 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544929+0000", 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001196522 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545004+0000", 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4962999999999994e-05 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545052+0000", 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8136999999999997e-05 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11779 3.9 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.885783064999998, 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020806181999999999, 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.538 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11779 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546926+0000", 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9769999999999999e-06 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552330+0000", 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054043980000000004 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552353+0000", 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2875999999999999e-05 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552362+0000", 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7299999999999994e-06 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552447+0000", 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5620999999999999e-05 2024-09-16T13:17:54.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552710+0000", 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026273300000000002 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.553446+0000", 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073563299999999997 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.559002+0000", 2024-09-16T13:17:54.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055564580000000002 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567615+0000", 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086126680000000004 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567689+0000", 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3839999999999998e-05 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567730+0000", 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1248000000000002e-05 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13736 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.134711887999998, 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018229967999999999, 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13736 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:54.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2e-07 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297998+0000", 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.824e-06 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298004+0000", 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8730000000000004e-06 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298008+0000", 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312651+0000", 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014643351000000001 2024-09-16T13:17:54.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312740+0000", 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8466000000000003e-05 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313081+0000", 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034067899999999998 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315982+0000", 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002900943 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316051+0000", 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9029999999999995e-05 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316101+0000", 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0680000000000003e-05 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316225+0000", 2024-09-16T13:17:54.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012359099999999999 2024-09-16T13:17:54.550 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.550 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.550 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.550 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13737 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:54.550 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:54.551 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.134694767999999, 2024-09-16T13:17:54.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018381208999999999, 2024-09-16T13:17:54.551 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.551 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.551 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13737 2024-09-16T13:17:54.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.552 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:54.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:54.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:17:54.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:17:54.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298015+0000", 2024-09-16T13:17:54.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6869999999999999e-06 2024-09-16T13:17:54.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298021+0000", 2024-09-16T13:17:54.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7799999999999997e-06 2024-09-16T13:17:54.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298027+0000", 2024-09-16T13:17:54.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.071e-06 2024-09-16T13:17:54.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313829+0000", 2024-09-16T13:17:54.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015801696 2024-09-16T13:17:54.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314019+0000", 2024-09-16T13:17:54.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018995700000000001 2024-09-16T13:17:54.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:54.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314379+0000", 2024-09-16T13:17:54.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036036500000000002 2024-09-16T13:17:54.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315988+0000", 2024-09-16T13:17:54.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016084949999999999 2024-09-16T13:17:54.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316290+0000", 2024-09-16T13:17:54.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000302732 2024-09-16T13:17:54.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316324+0000", 2024-09-16T13:17:54.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3937999999999999e-05 2024-09-16T13:17:54.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316393+0000", 2024-09-16T13:17:54.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8966000000000003e-05 2024-09-16T13:17:54.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.560 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.560 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13738 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:54.560 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.134675785999999, 2024-09-16T13:17:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021400311000000002, 2024-09-16T13:17:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13738 2024-09-16T13:17:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:54.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:17:54.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298032+0000", 2024-09-16T13:17:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7400000000000003e-07 2024-09-16T13:17:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298034+0000", 2024-09-16T13:17:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5770000000000001e-06 2024-09-16T13:17:54.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298040+0000", 2024-09-16T13:17:54.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7010000000000001e-06 2024-09-16T13:17:54.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298044+0000", 2024-09-16T13:17:54.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2069999999999997e-06 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315033+0000", 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016988970999999999 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315106+0000", 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2723000000000002e-05 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315428+0000", 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032191099999999997 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319249+0000", 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003821166 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319313+0000", 2024-09-16T13:17:54.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319347+0000", 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4591999999999997e-05 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319431+0000", 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3893000000000005e-05 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13739 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.134658274, 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021506517999999999, 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13739 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298049+0000", 2024-09-16T13:17:54.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298052+0000", 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.819e-06 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298058+0000", 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6960000000000002e-06 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298061+0000", 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7560000000000001e-06 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316465+0000", 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018403744 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316747+0000", 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028175200000000002 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.317694+0000", 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000947115 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319304+0000", 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016102029999999999 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319446+0000", 2024-09-16T13:17:54.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142008 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319490+0000", 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3690999999999999e-05 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319555+0000", 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5191000000000007e-05 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13740 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.134604791000001, 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023881870999999999, 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.574 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13740 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298103+0000", 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298106+0000", 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1559999999999999e-06 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298120+0000", 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4474999999999999e-05 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298124+0000", 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1699999999999999e-06 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319635+0000", 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021510129999999999 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319762+0000", 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012746400000000001 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.320225+0000", 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046264800000000002 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321629+0000", 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00140383 2024-09-16T13:17:54.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:54.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321682+0000", 2024-09-16T13:17:54.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3720000000000001e-05 2024-09-16T13:17:54.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:54.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321757+0000", 2024-09-16T13:17:54.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:17:54.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321984+0000", 2024-09-16T13:17:54.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022684199999999999 2024-09-16T13:17:54.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.580 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.580 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.580 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:54.582 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:54.582 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:54.582 INFO:teuthology.orchestra.run.smithi098.stdout:3947 write 0x526095a thru 0x52633ba (0x2a61 bytes) 2024-09-16T13:17:54.583 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:54.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:54.584 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:54.575+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:54.585 INFO:teuthology.orchestra.run.smithi098.stdout:3948 writesame 0xb0fd1bb thru 0xb1099da data_size 0x10(0xc820 bytes) 2024-09-16T13:17:54.585 INFO:teuthology.orchestra.run.smithi098.stdout:3949 read 0x10c126e thru 0x10c9788 (0x851b bytes) 2024-09-16T13:17:54.585 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.586 INFO:teuthology.orchestra.run.smithi098.stdout:3951 trunc from 0xb1099db to 0xcdcf110 2024-09-16T13:17:54.587 INFO:teuthology.orchestra.run.smithi098.stdout:3952 write 0x7b17c16 thru 0x7b269ef (0xedda bytes) 2024-09-16T13:17:54.588 INFO:teuthology.orchestra.run.smithi098.stdout:3956 punch from 0xb52ed51 to 0xb53c842, (0xdaf1 bytes) 2024-09-16T13:17:54.588 INFO:teuthology.orchestra.run.smithi098.stdout:3957 read 0x862f8b7 thru 0x8633c83 (0x43cd bytes) 2024-09-16T13:17:54.588 INFO:teuthology.orchestra.run.smithi098.stdout:3958 write 0xad83788 thru 0xad92f12 (0xf78b bytes) 2024-09-16T13:17:54.588 INFO:teuthology.orchestra.run.smithi098.stdout:3960 trunc from 0xcdcf110 to 0x38bc93e 2024-09-16T13:17:54.588 INFO:teuthology.orchestra.run.smithi098.stdout:3962 read 0x107c4d8 thru 0x1089804 (0xd32d bytes) 2024-09-16T13:17:54.596 INFO:teuthology.orchestra.run.smithi098.stdout:3965 trunc from 0x38bc93e to 0xd10030c 2024-09-16T13:17:54.611 INFO:teuthology.orchestra.run.smithi098.stdout:3966 write 0xa7ca92e thru 0xa7d648f (0xbb62 bytes) 2024-09-16T13:17:54.618 INFO:teuthology.orchestra.run.smithi098.stdout:3967 read 0xc12e65f thru 0xc1335da (0x4f7c bytes) 2024-09-16T13:17:54.618 INFO:teuthology.orchestra.run.smithi098.stdout:3968 writesame 0x2a350ba thru 0x2a35f2e data_size 0x1(0xe75 bytes) 2024-09-16T13:17:54.629 INFO:teuthology.orchestra.run.smithi098.stdout:3970 write 0xbb1bfcf thru 0xbb1cd89 (0xdbb bytes) 2024-09-16T13:17:54.636 INFO:teuthology.orchestra.run.smithi098.stdout:3971 read 0xa119d6 thru 0xa1cc76 (0xb2a1 bytes) 2024-09-16T13:17:54.652 INFO:teuthology.orchestra.run.smithi098.stdout:3972 writesame 0xe3d9807 thru 0xe3e44da data_size 0x12(0xacd4 bytes) 2024-09-16T13:17:54.652 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.652 INFO:teuthology.orchestra.run.smithi098.stdout:3974 writesame 0x9c590f6 thru 0x9c5fb66 data_size 0x1f(0x6a71 bytes) 2024-09-16T13:17:54.657 INFO:teuthology.orchestra.run.smithi098.stdout:3975 writesame 0x6bc328c thru 0x6bce13e data_size 0x1(0xaeb3 bytes) 2024-09-16T13:17:54.682 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:54.675+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:54.695 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:54.695 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:54.695 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:54.695 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:3976 read 0xa22bf60 thru 0xa22e74d (0x27ee bytes) 2024-09-16T13:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:3977 punch from 0x8eb6a1f to 0x8ebc4d5, (0x5ab6 bytes) 2024-09-16T13:17:54.704 INFO:teuthology.orchestra.run.smithi098.stdout:3978 writesame 0xa6e76ba thru 0xa6f1d7e data_size 0x6b(0xa6c5 bytes) 2024-09-16T13:17:54.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:54.732 INFO:teuthology.orchestra.run.smithi098.stdout:3979 trunc from 0xe3e44db to 0x61512cf 2024-09-16T13:17:54.786 INFO:teuthology.orchestra.run.smithi098.stdout:3980 write 0xbedb157 thru 0xbeddb39 (0x29e3 bytes) 2024-09-16T13:17:54.786 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:54.779+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:54.794 INFO:teuthology.orchestra.run.smithi098.stdout:3982 read 0xa91666f thru 0xa918516 (0x1ea8 bytes) 2024-09-16T13:17:54.794 INFO:teuthology.orchestra.run.smithi098.stdout:3983 write 0x3b74338 thru 0x3b7a361 (0x602a bytes) 2024-09-16T13:17:54.813 INFO:teuthology.orchestra.run.smithi098.stdout:3984 write 0xc6ced41 thru 0xc6d1b69 (0x2e29 bytes) 2024-09-16T13:17:54.821 INFO:teuthology.orchestra.run.smithi098.stdout:3985 write 0x983adad thru 0x9848ee3 (0xe137 bytes) 2024-09-16T13:17:54.828 INFO:teuthology.orchestra.run.smithi098.stdout:3986 punch from 0xa3220ed to 0xa327811, (0x5724 bytes) 2024-09-16T13:17:54.829 INFO:teuthology.orchestra.run.smithi098.stdout:3987 read 0xa6fb64e thru 0xa707eb7 (0xc86a bytes) 2024-09-16T13:17:54.829 INFO:teuthology.orchestra.run.smithi098.stdout:3988 write 0x88b2bb7 thru 0x88b6dbd (0x4207 bytes) 2024-09-16T13:17:54.836 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:54.837 INFO:teuthology.orchestra.run.smithi098.stdout:3990 read 0x4245b21 thru 0x425212a (0xc60a bytes) 2024-09-16T13:17:54.837 INFO:teuthology.orchestra.run.smithi098.stdout:3991 read 0xad95cbb thru 0xad9698c (0xcd2 bytes) 2024-09-16T13:17:54.837 INFO:teuthology.orchestra.run.smithi098.stdout:3992 writesame 0xad99f20 thru 0xada7f3a data_size 0x1(0xe01b bytes) 2024-09-16T13:17:54.865 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:54.865 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:54.865 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:54.866 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:54.866 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:54.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:54.885 INFO:teuthology.orchestra.run.smithi098.stdout:3993 write 0x4d28c4a thru 0x4d36f40 (0xe2f7 bytes) 2024-09-16T13:17:54.887 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:54.880+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:54.894 INFO:teuthology.orchestra.run.smithi098.stdout:3994 clone 24 order 23 su 32768 sc 6 2024-09-16T13:17:54.977 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2523 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 524288~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947385074000003, 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014015147, 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2523 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024654+0000", 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024656+0000", 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.131e-06 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024660+0000", 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6990000000000001e-06 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024663+0000", 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0290000000000001e-06 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038456+0000", 2024-09-16T13:17:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013793481 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038549+0000", 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2608000000000005e-05 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038669+0000", 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011982800000000001 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2524 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947371500000003, 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014381392, 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2524 2024-09-16T13:17:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024668+0000", 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8400000000000003e-07 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024670+0000", 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.849e-06 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024675+0000", 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.451e-06 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024678+0000", 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8739999999999999e-06 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038723+0000", 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014044562999999999 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038811+0000", 2024-09-16T13:17:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039049+0000", 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023761100000000001 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2525 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 557056~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947337152999999, 2024-09-16T13:17:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014602618, 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2525 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024705+0000", 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4650000000000001e-06 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024711+0000", 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024716+0000", 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1710000000000004e-06 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039096+0000", 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014380184000000001 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039187+0000", 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1065999999999999e-05 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039304+0000", 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117333 2024-09-16T13:17:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2526 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1507328~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947318983999999, 2024-09-16T13:17:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014823846, 2024-09-16T13:17:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2526 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024722+0000", 2024-09-16T13:17:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024729+0000", 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024733+0000", 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039337+0000", 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014604026000000001 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039427+0000", 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0437000000000002e-05 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039544+0000", 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116234 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2527 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1605632~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947301221000004, 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015041116, 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2527 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024738+0000", 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024746+0000", 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2130000000000008e-06 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024753+0000", 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:17:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024758+0000", 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7670000000000003e-06 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014820533 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039670+0000", 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1825999999999998e-05 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039779+0000", 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108424 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2528 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947275783000002, 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015245242000000001, 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2528 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024765+0000", 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1040000000000001e-06 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024772+0000", 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024776+0000", 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039810+0000", 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015034194000000001 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039897+0000", 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6864e-05 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040008+0000", 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110802 2024-09-16T13:17:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2529 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2555904~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947256664000001, 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015471574, 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2529 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024785+0000", 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5969999999999999e-06 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024800+0000", 2024-09-16T13:17:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4681999999999999e-05 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024806+0000", 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040039+0000", 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015233007999999999 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040143+0000", 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010322799999999999 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040254+0000", 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011092899999999999 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2530 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947226383999997, 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015667606000000001, 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2530 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024813+0000", 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024815+0000", 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5160000000000001e-06 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024827+0000", 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1915e-05 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024834+0000", 2024-09-16T13:17:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5180000000000001e-06 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040286+0000", 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015452565999999999 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040375+0000", 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040480+0000", 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105388 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2531 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3538944~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947199169000001, 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015867612999999999, 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2531 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024843+0000", 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6960000000000001e-06 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024856+0000", 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-05 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024863+0000", 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6069999999999996e-06 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040513+0000", 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650662999999999 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040599+0000", 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5439999999999995e-05 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040707+0000", 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108461 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2532 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947168664000003, 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061912000000001, 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2532 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024871+0000", 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-16T13:17:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024874+0000", 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024881+0000", 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0570000000000004e-06 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024886+0000", 2024-09-16T13:17:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040742+0000", 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015856808 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040824+0000", 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040932+0000", 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108378 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2533 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947148138999999, 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016272282999999998, 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2533 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024893+0000", 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419999999999999e-06 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024900+0000", 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024903+0000", 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040965+0000", 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061565999999999 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041052+0000", 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7213000000000007e-05 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041163+0000", 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011066899999999999 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2534 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3604480~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947129607999997, 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016482746999999999, 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2534 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024910+0000", 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1299999999999999e-07 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024912+0000", 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024920+0000", 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024924+0000", 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.036e-06 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041197+0000", 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016273242 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041282+0000", 2024-09-16T13:17:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5013999999999996e-05 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041392+0000", 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109567 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2535 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947108946, 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016691902000000002, 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2535 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7599999999999997e-07 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024932+0000", 2024-09-16T13:17:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024949+0000", 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7053000000000001e-05 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024961+0000", 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041425+0000", 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016463851000000002 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041515+0000", 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9549999999999995e-05 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041622+0000", 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107132 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2536 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1638400~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947071139999998, 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016880869999999999, 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2536 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024971+0000", 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.615e-06 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024988+0000", 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6912999999999999e-05 2024-09-16T13:17:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024993+0000", 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041653+0000", 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016660245000000001 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041743+0000", 2024-09-16T13:17:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0099999999999995e-05 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041848+0000", 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105404 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2537 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1671168~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947040387000001, 2024-09-16T13:17:55.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017075346000000002, 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2537 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024999+0000", 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025002+0000", 2024-09-16T13:17:55.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025009+0000", 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025013+0000", 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1840000000000001e-06 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041886+0000", 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016873249999999999 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041971+0000", 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4487000000000006e-05 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042074+0000", 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102847 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.046 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2538 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2621440~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.947020215999999, 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017314057000000001, 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2538 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.047 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025019+0000", 2024-09-16T13:17:55.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025021+0000", 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025033+0000", 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1985e-05 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025043+0000", 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3810000000000005e-06 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042107+0000", 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017064471000000001 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042203+0000", 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5810999999999995e-05 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042333+0000", 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012956099999999999 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2539 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.946992487999999, 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017500180000000001, 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.051 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2539 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025047+0000", 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025053+0000", 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025058+0000", 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025061+0000", 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.072e-06 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042365+0000", 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017304885999999998 2024-09-16T13:17:55.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042458+0000", 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2991000000000005e-05 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042546+0000", 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7904000000000004e-05 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:22787 3.17 3:edd0f01e:::rbd_object_map.112db5030c67:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.654607906000001, 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015269972, 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 22787 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317432+0000", 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.127e-06 2024-09-16T13:17:55.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317439+0000", 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0500000000000003e-06 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317458+0000", 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9326000000000001e-05 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317467+0000", 2024-09-16T13:17:55.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0669999999999998e-06 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329243+0000", 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011775551 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329293+0000", 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9709000000000001e-05 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329505+0000", 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021237200000000001 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.330969+0000", 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014635290000000001 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332590+0000", 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001621485 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332655+0000", 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4670000000000003e-05 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332701+0000", 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6085999999999999e-05 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.061 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:23429 3.5 3:a41c7f85:::rbd_object_map.112dc49cabe2:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.8829117390000008, 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014415534000000001, 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 23429 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089128+0000", 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7599999999999996e-07 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089140+0000", 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1939999999999999e-05 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089164+0000", 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4430000000000002e-05 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089173+0000", 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8200000000000003e-06 2024-09-16T13:17:55.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.097878+0000", 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087055320000000002 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.098035+0000", 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156537 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.098605+0000", 2024-09-16T13:17:55.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057031900000000001 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103250+0000", 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046448119999999999 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103351+0000", 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100763 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:55.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103472+0000", 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121091 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103542+0000", 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0513999999999996e-05 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:23430 3.5 3:a41c7f85:::rbd_object_map.112dc49cabe2:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T13:17:55.067 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.8828558490000002, 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016212550999999999, 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 23430 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089184+0000", 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089192+0000", 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8730000000000007e-06 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089210+0000", 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8127000000000001e-05 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089217+0000", 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7130000000000005e-06 2024-09-16T13:17:55.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.099775+0000", 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010557957999999999 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.099894+0000", 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011883999999999999 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.100458+0000", 2024-09-16T13:17:55.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056368200000000003 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103330+0000", 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028723690000000001 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105122+0000", 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017912430000000001 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:55.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105297+0000", 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175055 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105395+0000", 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8758000000000006e-05 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.073 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:55.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:55.075 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:55.064+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:55.176 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:55.168+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:55.186 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:55.187 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:55.187 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:17:55.187 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:55.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:55.253 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone23 from 0xc6d1b6a (overlap 0x13fd1c) to 0x3ff6c 2024-09-16T13:17:55.277 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:55.268+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:55.301 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:55.301 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:55.301 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:55.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:55.379 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:55.371+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:55.412 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:55.412 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:55.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:55.412 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:55.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11487 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 430080~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.920236137000003, 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014286111000000001, 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11487 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486860+0000", 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.04e-06 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486866+0000", 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0629999999999996e-06 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486881+0000", 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5023e-05 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486889+0000", 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8149999999999995e-06 2024-09-16T13:17:55.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.500906+0000", 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014016416 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501008+0000", 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102243 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501146+0000", 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013751099999999999 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11535 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.919553164, 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014337727999999999, 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.417 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11535 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487543+0000", 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023e-06 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487551+0000", 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0309999999999998e-06 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487566+0000", 2024-09-16T13:17:55.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5000999999999999e-05 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487578+0000", 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1066e-05 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501655+0000", 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014077151 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501747+0000", 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2670999999999994e-05 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501880+0000", 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013278500000000001 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11541 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 413696~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.919508614000002, 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014490381, 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11541 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487588+0000", 2024-09-16T13:17:55.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487592+0000", 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4920000000000004e-06 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487607+0000", 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4926000000000001e-05 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487616+0000", 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6829999999999997e-06 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501906+0000", 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0142898 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501959+0000", 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2985000000000002e-05 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502077+0000", 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118495 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11542 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 393216~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.919470089000001, 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014676434, 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.426 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11542 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487626+0000", 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.034e-06 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487631+0000", 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7659999999999998e-06 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487646+0000", 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4786e-05 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487655+0000", 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6489999999999997e-06 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502110+0000", 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014455749 2024-09-16T13:17:55.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502209+0000", 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8608000000000002e-05 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502302+0000", 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2842000000000003e-05 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11544 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 466944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.919429127999997, 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014846796000000001, 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11544 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487667+0000", 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.569999999999999e-07 2024-09-16T13:17:55.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487672+0000", 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2459999999999997e-06 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487686+0000", 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4847e-05 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487695+0000", 2024-09-16T13:17:55.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9109999999999999e-06 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502337+0000", 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014641861000000001 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502440+0000", 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010282599999999999 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502513+0000", 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3147999999999998e-05 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11547 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.919390000999996, 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015113645, 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.435 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11547 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:17:55.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487707+0000", 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.015e-06 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487711+0000", 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8289999999999998e-06 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487727+0000", 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5497000000000001e-05 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487736+0000", 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1409999999999994e-06 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502557+0000", 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014821212 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502675+0000", 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117741 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502819+0000", 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014421 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.439 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11549 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 413696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.919349578999999, 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016244525999999999, 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11549 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487747+0000", 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.094e-06 2024-09-16T13:17:55.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487752+0000", 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.758e-06 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487770+0000", 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8502999999999999e-05 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487779+0000", 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1579999999999994e-06 2024-09-16T13:17:55.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502868+0000", 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015088811000000001 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502916+0000", 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8109000000000003e-05 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503990+0000", 2024-09-16T13:17:55.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010740929999999999 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11578 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 409600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.919299137000003, 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015410814, 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.444 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11578 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487798+0000", 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.646e-06 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487804+0000", 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487821+0000", 2024-09-16T13:17:55.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7677999999999999e-05 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487830+0000", 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9090000000000006e-06 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503033+0000", 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015202337999999999 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503123+0000", 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0414000000000005e-05 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503207+0000", 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4172000000000007e-05 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11580 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:55.448 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.919255501000002, 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015544980999999999, 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11580 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:55.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487841+0000", 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487846+0000", 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3919999999999996e-06 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487861+0000", 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4946000000000001e-05 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487869+0000", 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8750000000000006e-06 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503240+0000", 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015370511999999999 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503334+0000", 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4265000000000006e-05 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503385+0000", 2024-09-16T13:17:55.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0773999999999999e-05 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11617 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 430080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.917935802999999, 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376203000000001, 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.453 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11617 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489162+0000", 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3089999999999998e-06 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489171+0000", 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3430000000000002e-06 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489198+0000", 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6449999999999999e-05 2024-09-16T13:17:55.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489216+0000", 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8057999999999999e-05 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503402+0000", 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014186196999999999 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503454+0000", 2024-09-16T13:17:55.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1989e-05 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503536+0000", 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1856999999999999e-05 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11624 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 503808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.457 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.917732715, 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376676, 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11624 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:55.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489365+0000", 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.795e-06 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489372+0000", 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4030000000000001e-06 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489392+0000", 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0129e-05 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489404+0000", 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1555e-05 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503558+0000", 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014154771999999999 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503644+0000", 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5691000000000003e-05 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503740+0000", 2024-09-16T13:17:55.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5330999999999999e-05 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11630 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.917679378000003, 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014893777, 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11630 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489417+0000", 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6800000000000009e-07 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489435+0000", 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7818000000000001e-05 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489450+0000", 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5072000000000001e-05 2024-09-16T13:17:55.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489459+0000", 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3009999999999995e-06 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503787+0000", 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01432719 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503889+0000", 2024-09-16T13:17:55.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102604 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504310+0000", 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042082399999999998 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11632 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.917626265000003, 2024-09-16T13:17:55.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01524066, 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11632 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489470+0000", 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.78e-07 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489475+0000", 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9400000000000001e-06 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489488+0000", 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3176999999999999e-05 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489497+0000", 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4919999999999993e-06 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504364+0000", 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014866846 2024-09-16T13:17:55.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504485+0000", 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012142999999999999 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504710+0000", 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000224897 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11639 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.917584494000003, 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015375941000000001, 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11639 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489512+0000", 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:17:55.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489517+0000", 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.938e-06 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489532+0000", 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.47e-05 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489540+0000", 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2949999999999994e-06 2024-09-16T13:17:55.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504765+0000", 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015225449 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504832+0000", 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6929999999999998e-05 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504887+0000", 2024-09-16T13:17:55.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4681e-05 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11641 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.917537983999999, 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015522312999999999, 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11641 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489559+0000", 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.066e-06 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489564+0000", 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9370000000000003e-06 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489579+0000", 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5466000000000001e-05 2024-09-16T13:17:55.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489588+0000", 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504925+0000", 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015337353 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505025+0000", 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9770000000000002e-05 2024-09-16T13:17:55.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505080+0000", 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4792e-05 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11667 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 16384~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.917177215000002, 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015353510000000001, 2024-09-16T13:17:55.479 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11667 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489920+0000", 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.84e-06 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489928+0000", 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6969999999999993e-06 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489953+0000", 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5154999999999999e-05 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489965+0000", 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2292e-05 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505115+0000", 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015149793999999999 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505184+0000", 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8858999999999999e-05 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505272+0000", 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7873000000000004e-05 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.483 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11672 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 36864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.917116810000003, 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015615053, 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11672 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489980+0000", 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489985+0000", 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490000+0000", 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490010+0000", 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.183e-06 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505335+0000", 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015325167000000001 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505471+0000", 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013581699999999999 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505594+0000", 2024-09-16T13:17:55.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123168 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11676 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.917076066, 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015779395000000002, 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.488 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11676 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490021+0000", 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.575e-06 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490026+0000", 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5170000000000001e-06 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490041+0000", 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4955e-05 2024-09-16T13:17:55.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490049+0000", 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5520000000000005e-06 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505647+0000", 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015597489000000001 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505743+0000", 2024-09-16T13:17:55.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6908000000000001e-05 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505799+0000", 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5399000000000003e-05 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11678 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:55.492 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.917036793000001, 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015992643000000001, 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11678 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:55.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490060+0000", 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3400000000000001e-06 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490065+0000", 2024-09-16T13:17:55.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4830000000000001e-06 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490079+0000", 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4854999999999999e-05 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490088+0000", 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1119999999999996e-06 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505849+0000", 2024-09-16T13:17:55.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015761718000000001 2024-09-16T13:17:55.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505966+0000", 2024-09-16T13:17:55.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011645399999999999 2024-09-16T13:17:55.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506051+0000", 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5680999999999995e-05 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11707 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 53248~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.916817664, 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016106869999999999, 2024-09-16T13:17:55.497 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11707 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490280+0000", 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7659999999999999e-06 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490287+0000", 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.503e-06 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490307+0000", 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9989000000000001e-05 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490318+0000", 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0967e-05 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506171+0000", 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015853367 2024-09-16T13:17:55.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506265+0000", 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3190999999999996e-05 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506385+0000", 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120087 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.501 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.502 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.502 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:55.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:55.503 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:55.495+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:55.560 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #22, image image_client.0-clone22 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent23 2024-09-16T13:17:55.580 INFO:teuthology.orchestra.run.smithi098.stdout:3995 write 0xb3a2a55 thru 0xb3a4e17 (0x23c3 bytes) 2024-09-16T13:17:55.603 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:55.595+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:55.607 INFO:teuthology.orchestra.run.smithi098.stdout:3997 writesame 0x16076dc thru 0x16123f1 data_size 0x69(0xad16 bytes) 2024-09-16T13:17:55.639 INFO:teuthology.orchestra.run.smithi098.stdout:3998 write 0x181b026 thru 0x181cdce (0x1da9 bytes) 2024-09-16T13:17:55.654 INFO:teuthology.orchestra.run.smithi098.stdout:4000 write 0xe7e9a2b thru 0xe7f8301 (0xe8d7 bytes) 2024-09-16T13:17:55.664 INFO:teuthology.orchestra.run.smithi098.stdout:4001 read 0xaae6910 thru 0xaaeef73 (0x8664 bytes) 2024-09-16T13:17:55.664 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:55.664 INFO:teuthology.orchestra.run.smithi098.stdout:4003 punch from 0x2604aa to 0x268890, (0x83e6 bytes) 2024-09-16T13:17:55.664 INFO:teuthology.orchestra.run.smithi098.stdout:4004 punch from 0x9befe99 to 0x9bf77eb, (0x7952 bytes) 2024-09-16T13:17:55.664 INFO:teuthology.orchestra.run.smithi098.stdout:4006 write 0xac5c1dd thru 0xac6abe8 (0xea0c bytes) 2024-09-16T13:17:55.668 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:55.668 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:55.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.668 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:28799 3.e 3.a03b7d4e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:17:55.668 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:55.663371+0000", 2024-09-16T13:17:55.668 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0015666409999999999, 2024-09-16T13:17:55.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015917240000000001, 2024-09-16T13:17:55.668 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.668 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:17:55.668 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 28799 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663371+0000", 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663371+0000", 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663372+0000", 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0399999999999995e-07 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663375+0000", 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4400000000000001e-06 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663400+0000", 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4267e-05 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663411+0000", 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1629e-05 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.664937+0000", 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015260639999999999 2024-09-16T13:17:55.671 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:28800 3.e 3.a03b7d4e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:55.663421+0000", 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.001517039, 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001613593, 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.672 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.673 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 28800 2024-09-16T13:17:55.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.673 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663421+0000", 2024-09-16T13:17:55.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663421+0000", 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663422+0000", 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9500000000000001e-07 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663427+0000", 2024-09-16T13:17:55.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8579999999999999e-06 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663443+0000", 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6798000000000001e-05 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663451+0000", 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5639999999999999e-06 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.675 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:28801 3.e 3.a03b7d4e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:55.663459+0000", 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0014785309999999999, 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016302160000000001, 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 28801 2024-09-16T13:17:55.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663459+0000", 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663459+0000", 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663460+0000", 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.71e-07 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663464+0000", 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1949999999999996e-06 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663481+0000", 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6741000000000001e-05 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663489+0000", 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5480000000000004e-06 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:28802 3.e 3.a03b7d4e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:55.663497+0000", 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.001440555, 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001648545, 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:55.679 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 28802 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663497+0000", 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663497+0000", 2024-09-16T13:17:55.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663498+0000", 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8199999999999999e-07 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663502+0000", 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7479999999999999e-06 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663513+0000", 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1487e-05 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663521+0000", 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.525e-06 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:28803 3.e 3.a03b7d4e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:17:55.682 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:55.663529+0000", 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.001409345, 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016728649999999999, 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 28803 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663529+0000", 2024-09-16T13:17:55.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663529+0000", 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663530+0000", 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1480000000000001e-06 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663533+0000", 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3739999999999998e-06 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663545+0000", 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1827999999999999e-05 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663551+0000", 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.426e-06 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.685 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.686 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:28804 3.e 3.a03b7d4e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:17:55.686 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:55.663559+0000", 2024-09-16T13:17:55.686 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0013789799999999999, 2024-09-16T13:17:55.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016927940000000001, 2024-09-16T13:17:55.686 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.686 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:55.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.687 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 28804 2024-09-16T13:17:55.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.687 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663559+0000", 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663559+0000", 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663560+0000", 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4700000000000001e-07 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663563+0000", 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.822e-06 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663575+0000", 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1093e-05 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663581+0000", 2024-09-16T13:17:55.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8449999999999997e-06 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:28805 3.e 3.a03b7d4e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:55.663589+0000", 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.001349294, 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001709885, 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.690 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 28805 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663589+0000", 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663589+0000", 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663589+0000", 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663593+0000", 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.709e-06 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663607+0000", 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3913999999999999e-05 2024-09-16T13:17:55.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:55.663615+0000", 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6539999999999999e-06 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 7 2024-09-16T13:17:55.693 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:55.694 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:55.694 INFO:teuthology.orchestra.run.smithi098.stdout:4008 read 0xd77453d thru 0xd782560 (0xe024 bytes) 2024-09-16T13:17:55.694 INFO:teuthology.orchestra.run.smithi098.stdout:4010 trunc from 0xe7f8302 to 0xa70aa61 2024-09-16T13:17:55.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:55.727 INFO:teuthology.orchestra.run.smithi098.stdout:4012 write 0x50aaffc thru 0x50b5339 (0xa33e bytes) 2024-09-16T13:17:55.740 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:55.740 INFO:teuthology.orchestra.run.smithi098.stdout:4015 read 0x18d0022 thru 0x18dc19b (0xc17a bytes) 2024-09-16T13:17:55.741 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:55.735+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:55.741 INFO:teuthology.orchestra.run.smithi098.stdout:4016 write 0x907da4d thru 0x9082b47 (0x50fb bytes) 2024-09-16T13:17:55.744 INFO:teuthology.orchestra.run.smithi098.stdout:4017 punch from 0x24af1e0 to 0x24b7995, (0x87b5 bytes) 2024-09-16T13:17:55.745 INFO:teuthology.orchestra.run.smithi098.stdout:4019 punch from 0x7e5d11f to 0x7e6520a, (0x80eb bytes) 2024-09-16T13:17:55.745 INFO:teuthology.orchestra.run.smithi098.stdout:4020 read 0x3550c30 thru 0x355e741 (0xdb12 bytes) 2024-09-16T13:17:55.745 INFO:teuthology.orchestra.run.smithi098.stdout:4021 read 0x82ec8d2 thru 0x82ed52d (0xc5c bytes) 2024-09-16T13:17:55.745 INFO:teuthology.orchestra.run.smithi098.stdout:4022 punch from 0x8f083bc to 0x8f10165, (0x7da9 bytes) 2024-09-16T13:17:55.745 INFO:teuthology.orchestra.run.smithi098.stdout:4023 read 0xe3d4a2 thru 0xe45c40 (0x879f bytes) 2024-09-16T13:17:55.746 INFO:teuthology.orchestra.run.smithi098.stdout:4024 read 0x7c54431 thru 0x7c639a4 (0xf574 bytes) 2024-09-16T13:17:55.746 INFO:teuthology.orchestra.run.smithi098.stdout:4025 write 0x9fe6cdc thru 0x9feaead (0x41d2 bytes) 2024-09-16T13:17:55.752 INFO:teuthology.orchestra.run.smithi098.stdout:4026 trunc from 0xa70aa61 to 0xba7324e 2024-09-16T13:17:55.780 INFO:teuthology.orchestra.run.smithi098.stdout:4027 writesame 0xe73be96 thru 0xe747ea2 data_size 0x1(0xc00d bytes) 2024-09-16T13:17:55.780 INFO:teuthology.orchestra.run.smithi098.stdout:4029 punch from 0xf5e3bc to 0xf6d48c, (0xf0d0 bytes) 2024-09-16T13:17:55.780 INFO:teuthology.orchestra.run.smithi098.stdout:4030 clone 25 order 21 su 524288 sc 4 2024-09-16T13:17:55.793 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:55.794 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:17:55.794 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:17:55.794 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:17:55.794 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:55.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:55.804 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:55.796+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:55.903 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:17:55.903 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.87227111, 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:55.904 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:17:55.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:17:55.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.909 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.872123704000003, 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:17:55.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:17:55.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.915 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.872096361000004, 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:17:55.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:17:55.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:55.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:55.921 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.313421098999996, 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:17:55.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:55.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:55.927 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.313333506999996, 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:55.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:17:55.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.313163584999998, 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:17:55.933 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:17:55.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:17:55.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2259 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3014656~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161835009000001, 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016650893999999999, 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.939 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2259 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735831+0000", 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9530000000000005e-06 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735836+0000", 2024-09-16T13:17:55.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7450000000000003e-06 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735839+0000", 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.915e-06 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752296+0000", 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016457241000000001 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752360+0000", 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4108000000000004e-05 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752473+0000", 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112305 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2260 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.943 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161815087999997, 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016799073000000001, 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2260 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:55.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735843+0000", 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3480000000000001e-06 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735847+0000", 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5990000000000002e-06 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735849+0000", 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5859999999999999e-06 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752509+0000", 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016659377 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752559+0000", 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9774000000000002e-05 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752641+0000", 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2057000000000004e-05 2024-09-16T13:17:55.947 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2261 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161802899000001, 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016960698, 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.948 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2261 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.03e-07 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735855+0000", 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735860+0000", 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1110000000000001e-06 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735862+0000", 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752695+0000", 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01683343 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752744+0000", 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8414000000000001e-05 2024-09-16T13:17:55.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752814+0000", 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0623000000000005e-05 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2262 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1048576~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161792284999997, 2024-09-16T13:17:55.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017224107999999998, 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2262 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735865+0000", 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.91e-07 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735866+0000", 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1480000000000001e-06 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735870+0000", 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735877+0000", 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4370000000000001e-06 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752871+0000", 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016994061000000001 2024-09-16T13:17:55.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752962+0000", 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0470000000000006e-05 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753089+0000", 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126642 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2263 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161776826000001, 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017392088999999999, 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2263 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-07 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735881+0000", 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735885+0000", 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4180000000000001e-06 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735887+0000", 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-06 2024-09-16T13:17:55.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753116+0000", 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017228963 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753184+0000", 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7374999999999998e-05 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753272+0000", 2024-09-16T13:17:55.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8323000000000001e-05 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2264 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2064384~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161766974999999, 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017630541999999999, 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.961 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2264 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735891+0000", 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735895+0000", 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735897+0000", 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4669999999999998e-06 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753316+0000", 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017418685 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753403+0000", 2024-09-16T13:17:55.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7590000000000007e-05 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753520+0000", 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116863 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2265 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1081344~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:55.965 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161757092999999, 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017739263000000002, 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2265 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735901+0000", 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-09-16T13:17:55.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735904+0000", 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.304e-06 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735907+0000", 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3599999999999999e-06 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753534+0000", 2024-09-16T13:17:55.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017627446000000001 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753576+0000", 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1474999999999998e-05 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753639+0000", 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3188999999999994e-05 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2266 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2097152~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161747859999998, 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017970893000000002, 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2266 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735914+0000", 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1699999999999996e-06 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735918+0000", 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6859999999999999e-06 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735920+0000", 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4870000000000001e-06 2024-09-16T13:17:55.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753685+0000", 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017764582000000001 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753771+0000", 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6259000000000002e-05 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753880+0000", 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010844300000000001 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2267 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161733959000003, 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018110804000000001, 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2267 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735924+0000", 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735928+0000", 2024-09-16T13:17:55.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.416e-06 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735930+0000", 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753898+0000", 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017967852999999999 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753955+0000", 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6662000000000001e-05 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754034+0000", 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8904000000000002e-05 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2268 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3080192~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.978 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161724474000003, 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018252866, 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2268 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735934+0000", 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0950000000000001e-06 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735938+0000", 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735941+0000", 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8750000000000002e-06 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754071+0000", 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018129941 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754120+0000", 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8359000000000002e-05 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754185+0000", 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5358999999999995e-05 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2269 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161713095000003, 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018533516, 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2269 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:17:55.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735945+0000", 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1149999999999999e-06 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735948+0000", 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5250000000000001e-06 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735951+0000", 2024-09-16T13:17:55.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9909999999999998e-06 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754270+0000", 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018318081999999999 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754325+0000", 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5648e-05 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754477+0000", 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015188199999999999 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2270 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1114112~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161702626, 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019103282999999999, 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.987 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2270 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:17:55.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735955+0000", 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735956+0000", 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.096e-06 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735959+0000", 2024-09-16T13:17:55.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5140000000000002e-06 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735963+0000", 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.053e-06 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754822+0000", 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018858883 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754941+0000", 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118789 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755057+0000", 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116272 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2271 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1146880~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161680814, 2024-09-16T13:17:55.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019332309999999998, 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2271 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3500000000000002e-07 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735982+0000", 2024-09-16T13:17:55.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735987+0000", 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1930000000000003e-06 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735993+0000", 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4659999999999996e-06 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755081+0000", 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019088187 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755194+0000", 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011223299999999999 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755308+0000", 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114663 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:55.995 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2272 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 163840~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.161658983999999, 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019476291, 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2272 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.996 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:17:55.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:17:55.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:17:55.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736003+0000", 2024-09-16T13:17:55.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2279999999999998e-06 2024-09-16T13:17:55.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:17:55.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736007+0000", 2024-09-16T13:17:55.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.506e-06 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736011+0000", 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9280000000000003e-06 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755336+0000", 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019325189999999999 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:17:55.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755393+0000", 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7293999999999997e-05 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755474+0000", 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0635000000000003e-05 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:17:56.000 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:17:56.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:56.002 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:55.996+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:56.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:17:56.102 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:56.095+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:56.149 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:56.149 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:56.149 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:56.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:56.203 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:56.195+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:56.256 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:56.256 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:56.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:56.267 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone24 from 0xe747ea3 (overlap 0xa70aa61) to 0x74eddaa 2024-09-16T13:17:56.311 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:56.304+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11500 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 356352~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.864046068, 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014537983000000001, 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.366 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11500 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496474+0000", 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.801e-06 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496480+0000", 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1140000000000002e-06 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496484+0000", 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8770000000000003e-06 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507573+0000", 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011088473 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507624+0000", 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1591999999999997e-05 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511010+0000", 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033857380000000001 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.370 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11501 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 430080~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.864029701, 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014726846, 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11501 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496489+0000", 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3700000000000001e-07 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496491+0000", 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9570000000000001e-06 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496497+0000", 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2550000000000003e-06 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496501+0000", 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8639999999999998e-06 2024-09-16T13:17:56.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511060+0000", 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014559462 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511125+0000", 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4399999999999993e-05 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511215+0000", 2024-09-16T13:17:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0470999999999995e-05 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11530 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 339968~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.863095512999998, 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014141661, 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.375 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11530 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497423+0000", 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-07 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497434+0000", 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1195e-05 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497442+0000", 2024-09-16T13:17:56.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.323e-06 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497446+0000", 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1049999999999997e-06 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511275+0000", 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013828872000000001 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511389+0000", 2024-09-16T13:17:56.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114493 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511564+0000", 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017508700000000001 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11533 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 413696~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.863066429, 2024-09-16T13:17:56.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014296237999999999, 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11533 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.51e-07 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497458+0000", 2024-09-16T13:17:56.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9139999999999996e-06 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497464+0000", 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0469999999999999e-06 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497467+0000", 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5030000000000002e-06 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511593+0000", 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014125692 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511669+0000", 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5950000000000003e-05 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511748+0000", 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8681000000000001e-05 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11551 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 335872~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.863010402, 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014415314, 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11551 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.384 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497510+0000", 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.012e-06 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497516+0000", 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7880000000000003e-06 2024-09-16T13:17:56.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497524+0000", 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9049999999999997e-06 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511769+0000", 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245809 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511818+0000", 2024-09-16T13:17:56.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8092e-05 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511923+0000", 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010532299999999999 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11553 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 413696~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:56.388 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.862987552999996, 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014561233999999999, 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11553 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497531+0000", 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1899999999999998e-07 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497533+0000", 2024-09-16T13:17:56.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9149999999999999e-06 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497539+0000", 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.088e-06 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497550+0000", 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0834e-05 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511962+0000", 2024-09-16T13:17:56.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411947 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512018+0000", 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6755999999999997e-05 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512092+0000", 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3275000000000006e-05 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11577 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 409600~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.860880309000002, 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013920979999999999, 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11577 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:17:56.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499640+0000", 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1430000000000001e-06 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499646+0000", 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1029999999999998e-06 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499650+0000", 2024-09-16T13:17:56.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6370000000000001e-06 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513442+0000", 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013791805000000001 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513494+0000", 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2074999999999999e-05 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513559+0000", 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4874000000000003e-05 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11637 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.859987877999998, 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013965546000000001, 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.397 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11637 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500531+0000", 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2699999999999999e-07 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500534+0000", 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.055e-06 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500542+0000", 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6389999999999998e-06 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500553+0000", 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0974e-05 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514249+0000", 2024-09-16T13:17:56.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013696086 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514308+0000", 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8269000000000001e-05 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514496+0000", 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018799600000000001 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11638 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.859958396000003, 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014058684, 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11638 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.402 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:56.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500563+0000", 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0790000000000001e-06 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500580+0000", 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6472000000000001e-05 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500586+0000", 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2199999999999997e-06 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514522+0000", 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013935774 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514564+0000", 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2549000000000002e-05 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514618+0000", 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3942e-05 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11642 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.859873563000001, 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014093902, 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.406 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11642 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:56.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500645+0000", 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5200000000000002e-07 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500656+0000", 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1372999999999999e-05 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500664+0000", 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9019999999999999e-06 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500677+0000", 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2954e-05 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514649+0000", 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013971417 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514692+0000", 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3154000000000001e-05 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514738+0000", 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6650000000000002e-05 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.410 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11646 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.859835300999997, 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245793, 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11646 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500685+0000", 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2019999999999998e-06 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500698+0000", 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.279e-05 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500702+0000", 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7629999999999998e-06 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514772+0000", 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014070384999999999 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514865+0000", 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2980999999999996e-05 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514928+0000", 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3114000000000006e-05 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11648 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.859812171000002, 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411604999999999, 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11648 2024-09-16T13:17:56.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9900000000000001e-07 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500713+0000", 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.793e-06 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500719+0000", 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500722+0000", 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3809999999999999e-06 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514973+0000", 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014250924 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515031+0000", 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7864e-05 2024-09-16T13:17:56.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515117+0000", 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6697e-05 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11654 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.859445336999997, 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014438998, 2024-09-16T13:17:56.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11654 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8699999999999995e-07 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501075+0000", 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501082+0000", 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.844e-06 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501086+0000", 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7749999999999999e-06 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515152+0000", 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014065891000000001 2024-09-16T13:17:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515232+0000", 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0461000000000001e-05 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515512+0000", 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027934599999999998 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11675 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.858982742999999, 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014239586, 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11675 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501536+0000", 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:56.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501539+0000", 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3529999999999999e-06 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501546+0000", 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.109e-06 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501551+0000", 2024-09-16T13:17:56.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5109999999999997e-06 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515586+0000", 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014035334999999999 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515689+0000", 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010244200000000001 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515775+0000", 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6150999999999996e-05 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11679 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.858932819000003, 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014462922, 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.428 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11679 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:56.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501586+0000", 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501588+0000", 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3760000000000002e-06 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501594+0000", 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9950000000000002e-06 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501598+0000", 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7239999999999998e-06 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515840+0000", 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014242546999999999 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515949+0000", 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108591 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516048+0000", 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9123000000000006e-05 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11686 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.432 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.858916247000003, 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014648357000000001, 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11686 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:56.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501604+0000", 2024-09-16T13:17:56.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.846e-06 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501609+0000", 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3630000000000002e-06 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501613+0000", 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3749999999999999e-06 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516077+0000", 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014464351 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516173+0000", 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5742000000000006e-05 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516250+0000", 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7312999999999997e-05 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.436 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11689 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.858901682999999, 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014770922000000001, 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11689 2024-09-16T13:17:56.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501617+0000", 2024-09-16T13:17:56.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501618+0000", 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6780000000000001e-06 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501624+0000", 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3750000000000002e-06 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501627+0000", 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5149999999999998e-06 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516279+0000", 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014651928999999999 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516336+0000", 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6724000000000001e-05 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516387+0000", 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1334e-05 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11778 3.1 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.841567554999997, 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022495198000000001, 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11778 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518952+0000", 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.711e-06 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522261+0000", 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033087429999999998 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522296+0000", 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5335e-05 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522306+0000", 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0074999999999999e-05 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522379+0000", 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2554999999999999e-05 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522572+0000", 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019323100000000001 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.523266+0000", 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069379000000000001 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.534086+0000", 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010820112999999999 2024-09-16T13:17:56.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541312+0000", 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072264440000000003 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541394+0000", 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1710000000000002e-05 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541446+0000", 2024-09-16T13:17:56.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1490999999999999e-05 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:25474 3.6 3:6380c99b:::rbd_object_map.112de78841b1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:49.832331+0000", 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.5281867800000004, 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014819143999999999, 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 25474 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832331+0000", 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832331+0000", 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832332+0000", 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.15e-07 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832334+0000", 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.148e-06 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832337+0000", 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832340+0000", 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4760000000000001e-06 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.842963+0000", 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010623886000000001 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.843120+0000", 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015636400000000001 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.843657+0000", 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000536887 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846702+0000", 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030451300000000001 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.847054+0000", 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035228400000000002 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.847116+0000", 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1977000000000006e-05 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.847150+0000", 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4257000000000001e-05 2024-09-16T13:17:56.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:25567 3.6 3:6380c99b:::rbd_object_map.112de78841b1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.3778062709999999, 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014520869, 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:56.453 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 25567 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7100000000000002e-07 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982715+0000", 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0639999999999998e-06 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982736+0000", 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0421999999999999e-05 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982743+0000", 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2509999999999997e-06 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.990894+0000", 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0081508239999999992 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.991014+0000", 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011978599999999999 2024-09-16T13:17:56.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.991456+0000", 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044261099999999998 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.992802+0000", 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001345398 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.997165+0000", 2024-09-16T13:17:56.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043631399999999997 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.997200+0000", 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5474e-05 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.997233+0000", 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2327999999999999e-05 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:56.459 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:56.459 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:56.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:56.460 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:56.452+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:56.561 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:56.555+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:56.570 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:56.570 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:56.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:56.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:17:56.633 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #23, image image_client.0-clone23 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent24 2024-09-16T13:17:56.662 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:56.652+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:56.663 INFO:teuthology.orchestra.run.smithi098.stdout:4031 trunc from 0xe747ea3 to 0x2211177 2024-09-16T13:17:56.685 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:56.686 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:56.686 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:56.691 INFO:teuthology.orchestra.run.smithi098.stdout:4033 read 0xb1dc15 thru 0xb25610 (0x79fc bytes) 2024-09-16T13:17:56.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:56.710 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:56.715 INFO:teuthology.orchestra.run.smithi098.stdout:4034 writesame 0xbc5a8a5 thru 0xbc620b6 data_size 0x6d(0x7812 bytes) 2024-09-16T13:17:56.715 INFO:teuthology.orchestra.run.smithi098.stdout:4035 read 0x3f1bcfb thru 0x3f23484 (0x778a bytes) 2024-09-16T13:17:56.715 INFO:teuthology.orchestra.run.smithi098.stdout:4036 write 0x51b06e9 thru 0x51b6f2c (0x6844 bytes) 2024-09-16T13:17:56.721 INFO:teuthology.orchestra.run.smithi098.stdout:4038 writesame 0x66cbbd3 thru 0x66cdab2 data_size 0x34(0x1ee0 bytes) 2024-09-16T13:17:56.727 INFO:teuthology.orchestra.run.smithi098.stdout:4039 read 0x4a28df2 thru 0x4a2c80a (0x3a19 bytes) 2024-09-16T13:17:56.727 INFO:teuthology.orchestra.run.smithi098.stdout:4040 read 0x9966f7a thru 0x9973680 (0xc707 bytes) 2024-09-16T13:17:56.727 INFO:teuthology.orchestra.run.smithi098.stdout:4041 write 0xeb1802 thru 0xeb18df (0xde bytes) 2024-09-16T13:17:56.730 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:56.737 INFO:teuthology.orchestra.run.smithi098.stdout:4042 write 0x865009c thru 0x865715e (0x70c3 bytes) 2024-09-16T13:17:56.744 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:56.744 INFO:teuthology.orchestra.run.smithi098.stdout:4044 write 0xb3cdd74 thru 0xb3d33f9 (0x5686 bytes) 2024-09-16T13:17:56.748 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:56.751 INFO:teuthology.orchestra.run.smithi098.stdout:4045 punch from 0xb7bef16 to 0xb7cd420, (0xe50a bytes) 2024-09-16T13:17:56.751 INFO:teuthology.orchestra.run.smithi098.stdout:4048 writesame 0x729ee4a thru 0x72ac1ce data_size 0x1(0xd385 bytes) 2024-09-16T13:17:56.763 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:56.755+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:56.766 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:56.790 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:56.795 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:56.795 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:56.802 INFO:teuthology.orchestra.run.smithi098.stdout:4049 punch from 0x5d4b4ca to 0x5d4f095, (0x3bcb bytes) 2024-09-16T13:17:56.802 INFO:teuthology.orchestra.run.smithi098.stdout:4050 write 0x1f6aff4 thru 0x1f7605e (0xb06b bytes) 2024-09-16T13:17:56.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:56.815 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:56.816 INFO:teuthology.orchestra.run.smithi098.stdout:4051 read 0xa74f625 thru 0xa7550e8 (0x5ac4 bytes) 2024-09-16T13:17:56.829 INFO:teuthology.orchestra.run.smithi098.stdout:4052 writesame 0xe0c525d thru 0xe0c8699 data_size 0x1(0x343d bytes) 2024-09-16T13:17:56.829 INFO:teuthology.orchestra.run.smithi098.stdout:4053 writesame 0x6423509 thru 0x6424161 data_size 0x1d(0xc59 bytes) 2024-09-16T13:17:56.835 INFO:teuthology.orchestra.run.smithi098.stdout:4054 read 0x5c42151 thru 0x5c445ff (0x24af bytes) 2024-09-16T13:17:56.849 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:56.854 INFO:teuthology.orchestra.run.smithi098.stdout:4056 trunc from 0xe0c869a to 0x3ec41eb 2024-09-16T13:17:56.864 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:56.855+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:56.875 INFO:teuthology.orchestra.run.smithi098.stdout:4057 read 0x21f55d3 thru 0x22045d3 (0xf001 bytes) 2024-09-16T13:17:56.876 INFO:teuthology.orchestra.run.smithi098.stdout:4058 read 0x1802179 thru 0x1810abe (0xe946 bytes) 2024-09-16T13:17:56.877 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:56.877 INFO:teuthology.orchestra.run.smithi098.stdout:4062 read 0x1abdc9f thru 0x1ac0da0 (0x3102 bytes) 2024-09-16T13:17:56.877 INFO:teuthology.orchestra.run.smithi098.stdout:4063 punch from 0xb29730 to 0xb3195b, (0x822b bytes) 2024-09-16T13:17:56.882 INFO:teuthology.orchestra.run.smithi098.stdout:4064 trunc from 0x3ec41eb to 0x6558cfc 2024-09-16T13:17:56.883 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:56.885 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:56.886 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:56.901 INFO:teuthology.orchestra.run.smithi098.stdout:4065 trunc from 0x6558cfc to 0xcad6881 2024-09-16T13:17:56.910 INFO:teuthology.orchestra.run.smithi098.stdout:4066 read 0x2d3c3a1 thru 0x2d43d12 (0x7972 bytes) 2024-09-16T13:17:56.910 INFO:teuthology.orchestra.run.smithi098.stdout:4067 write 0xa626c0d thru 0xa635efc (0xf2f0 bytes) 2024-09-16T13:17:56.916 INFO:teuthology.orchestra.run.smithi098.stdout:4069 read 0x4eaa801 thru 0x4eb7ac2 (0xd2c2 bytes) 2024-09-16T13:17:56.921 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:56.921 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:56.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:17:56.935 INFO:teuthology.orchestra.run.smithi098.stdout:4070 trunc from 0xcad6881 to 0x156a491 2024-09-16T13:17:56.965 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:56.956+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:56.978 INFO:teuthology.orchestra.run.smithi098.stdout:4071 write 0x1909b00 thru 0x1915e35 (0xc336 bytes) 2024-09-16T13:17:56.984 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:56.984 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:56.984 INFO:teuthology.orchestra.run.smithi098.stdout:4075 punch from 0x187ec67 to 0x1887237, (0x85d0 bytes) 2024-09-16T13:17:56.984 INFO:teuthology.orchestra.run.smithi098.stdout:4076 trunc from 0x1915e36 to 0x12da2df 2024-09-16T13:17:57.038 INFO:teuthology.orchestra.run.smithi098.stdout:4077 write 0xeb7f648 thru 0xeb8a8a5 (0xb25e bytes) 2024-09-16T13:17:57.044 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:57.044 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:57.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:17:57.061 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:17:57.066 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:57.059+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:57.068 INFO:teuthology.orchestra.run.smithi098.stdout:4080 trunc from 0xeb8a8a6 to 0x2cb7d32 2024-09-16T13:17:57.127 INFO:teuthology.orchestra.run.smithi098.stdout:4081 write 0xcee13d6 thru 0xcee5e84 (0x4aaf bytes) 2024-09-16T13:17:57.133 INFO:teuthology.orchestra.run.smithi098.stdout:4082 read 0xed8955 thru 0xedf389 (0x6a35 bytes) 2024-09-16T13:17:57.134 INFO:teuthology.orchestra.run.smithi098.stdout:4083 punch from 0x2ad3b7c to 0x2ad8b66, (0x4fea bytes) 2024-09-16T13:17:57.144 INFO:teuthology.orchestra.run.smithi098.stdout:4084 trunc from 0xcee5e85 to 0x64a7ba3 2024-09-16T13:17:57.161 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:57.162 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:57.162 INFO:teuthology.orchestra.run.smithi098.stdout:4085 write 0x40d107 thru 0x4110bd (0x3fb7 bytes) 2024-09-16T13:17:57.166 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:57.156+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:57.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:17:57.182 INFO:teuthology.orchestra.run.smithi098.stdout:4087 read 0x38a7ace thru 0x38b0859 (0x8d8c bytes) 2024-09-16T13:17:57.189 INFO:teuthology.orchestra.run.smithi098.stdout:4088 trunc from 0x64a7ba3 to 0x28a5832 2024-09-16T13:17:57.217 INFO:teuthology.orchestra.run.smithi098.stdout:4089 write 0x9d53f73 thru 0x9d55bbc (0x1c4a bytes) 2024-09-16T13:17:57.223 INFO:teuthology.orchestra.run.smithi098.stdout:4090 writesame 0xfaf70c thru 0xfbdee0 data_size 0x49(0xe7d5 bytes) 2024-09-16T13:17:57.239 INFO:teuthology.orchestra.run.smithi098.stdout:4091 punch from 0x30d0ab5 to 0x30d258e, (0x1ad9 bytes) 2024-09-16T13:17:57.239 INFO:teuthology.orchestra.run.smithi098.stdout:4092 read 0x7377b99 thru 0x7382c6e (0xb0d6 bytes) 2024-09-16T13:17:57.239 INFO:teuthology.orchestra.run.smithi098.stdout:4095 read 0x9542a72 thru 0x9549865 (0x6df4 bytes) 2024-09-16T13:17:57.239 INFO:teuthology.orchestra.run.smithi098.stdout:4097 write 0x4e20971 thru 0x4e292a6 (0x8936 bytes) 2024-09-16T13:17:57.246 INFO:teuthology.orchestra.run.smithi098.stdout:4099 write 0x682dece thru 0x68323c9 (0x44fc bytes) 2024-09-16T13:17:57.252 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:57.252 INFO:teuthology.orchestra.run.smithi098.stdout:4101 punch from 0x8dd4b62 to 0x8de02dd, (0xb77b bytes) 2024-09-16T13:17:57.265 INFO:teuthology.orchestra.run.smithi098.stdout:4102 writesame 0xb7295ec thru 0xb72af80 data_size 0x25(0x1995 bytes) 2024-09-16T13:17:57.266 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:57.266 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:57.267 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:57.260+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:57.277 INFO:teuthology.orchestra.run.smithi098.stdout:4106 trunc from 0xb72af81 to 0x3ef8c28 2024-09-16T13:17:57.278 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:57.278 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:57.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:17:57.315 INFO:teuthology.orchestra.run.smithi098.stdout:4107 writesame 0x7a0ccfc thru 0x7a0d651 data_size 0x2(0x956 bytes) 2024-09-16T13:17:57.315 INFO:teuthology.orchestra.run.smithi098.stdout:4108 read 0x66bc16a thru 0x66bcacb (0x962 bytes) 2024-09-16T13:17:57.315 INFO:teuthology.orchestra.run.smithi098.stdout:4109 writesame 0x67b1de8 thru 0x67c0cfd data_size 0x65(0xef16 bytes) 2024-09-16T13:17:57.322 INFO:teuthology.orchestra.run.smithi098.stdout:4110 read 0x1d9a36b thru 0x1d9ee3d (0x4ad3 bytes) 2024-09-16T13:17:57.323 INFO:teuthology.orchestra.run.smithi098.stdout:4111 punch from 0x559f9f1 to 0x55a2305, (0x2914 bytes) 2024-09-16T13:17:57.323 INFO:teuthology.orchestra.run.smithi098.stdout:4112 writesame 0x576f40b thru 0x577764f data_size 0x1(0x8245 bytes) 2024-09-16T13:17:57.368 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:57.360+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:57.369 INFO:teuthology.orchestra.run.smithi098.stdout:4113 writesame 0x9e8f79e thru 0x9e972df data_size 0x9(0x7b42 bytes) 2024-09-16T13:17:57.369 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:57.369 INFO:teuthology.orchestra.run.smithi098.stdout:4115 write 0x17aad57 thru 0x17b6d42 (0xbfec bytes) 2024-09-16T13:17:57.372 INFO:teuthology.orchestra.run.smithi098.stdout:4116 write 0x7f8a1df thru 0x7f923ec (0x820e bytes) 2024-09-16T13:17:57.377 INFO:teuthology.orchestra.run.smithi098.stdout:4118 clone 26 order 21 su 65536 sc 3 2024-09-16T13:17:57.390 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:57.391 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:57.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:17:57.469 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:57.463+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:57.550 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:57.550 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:57.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:17:57.605 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:57.599+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:57.667 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:57.667 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:57.670 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:57.663+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:57.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:17:57.772 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:57.768+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:57.775 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:57.776 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:57.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:17:57.813 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:17:57.814 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:17:57.814 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T13:17:57.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T13:17:57.873 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:57.867+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:57.886 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:57.886 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:57.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:17:57.974 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:17:57.967+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:17:57.994 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:57.994 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:58.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:17:58.075 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:58.068+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:58.103 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:58.104 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:58.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:17:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:17:58.120 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 72, 2024-09-16T13:17:58.120 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "6ddcae89-6c18-4eb7-8712-1c53f3b07278", 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-16T13:16:23.176689+0000", 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-16T13:17:58.057079+0000", 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "last_up_change": "2024-09-16T13:16:33.765413+0000", 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "last_in_change": "2024-09-16T13:17:42.548353+0000", 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_num": 5804032, 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_set": [ 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nodeep-scrub", 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "pglog_hardlimit", 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "purged_snapdirs", 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_deletes", 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "sortbitwise" 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:58.121 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_version": 5, 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_snapshot": "", 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_max": 3, 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd": 8, 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "require_min_compat_client": "luminous", 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "min_compat_client": "luminous", 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "require_osd_release": "quincy", 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "pools": [ 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 1, 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": ".mgr", 2024-09-16T13:17:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:33.497561+0000", 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool", 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 1, 2024-09-16T13:17:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 1, 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 1, 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 1, 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 1, 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "17", 2024-09-16T13:17:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 0, 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 0, 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:17:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:17:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "options": { 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_max": 32, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_min": 1 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": {} 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 2, 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": "rbd", 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:40.994538+0000", 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 8193, 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:17:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 8, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 8, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 8, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 8, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 8, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:17:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "21", 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 3, 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 21, 2024-09-16T13:17:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:17:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:17:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "options": {}, 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd": {} 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "pool": 3, 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": "pool_client.0", 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "create_time": "2024-09-16T13:16:50.759705+0000", 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 8193, 2024-09-16T13:17:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": 1, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 2, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": 1, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_rule": 0, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_count": 0, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_target": 0, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "object_hash": 2, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscale_mode": "off", 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": 32, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num": 32, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_placement_num_target": 32, 2024-09-16T13:17:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_target": 32, 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num_pending": 32, 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "last_pg_merge_meta": { 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "source_pgid": "0.0", 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "ready_epoch": 0, 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_started": 0, 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "last_epoch_clean": 0, 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "source_version": "0'0", 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "target_version": "0'0" 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "last_change": "72", 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend": "0", 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T13:17:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "auid": 0, 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_mode": "selfmanaged", 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_seq": 29, 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_epoch": 72, 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_snaps": [], 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps": "[]", 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_bytes": 0, 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "quota_max_objects": 0, 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tiers": [], 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_of": -1, 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "read_tier": -1, 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "write_tier": -1, 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_mode": "none", 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_bytes": 0, 2024-09-16T13:17:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "target_max_objects": 0, 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_flush_age": 0, 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "cache_min_evict_age": 0, 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profile": "", 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_params": { 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "none" 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_period": 0, 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_count": 0, 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "use_gmt_hitset": true, 2024-09-16T13:17:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "min_read_recency_for_promote": 0, 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "min_write_recency_for_promote": 0, 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "hit_set_search_last_n": 0, 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "grade_table": [], 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "stripe_width": 0, 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "expected_num_objects": 0, 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "fast_read": false, 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "options": {}, 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "application_metadata": { 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd": {} 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "osds": [ 2024-09-16T13:17:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 0, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "be8543b4-3d28-4b37-a4ce-d2a56a09aeaf", 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0.8320770263671875, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 61, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6808", 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6809", 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6810", 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6811", 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6816", 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6817", 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6813", 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6814", 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22271 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6809/22271", 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6811/22271", 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6817/22271", 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6814/22271", 2024-09-16T13:17:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 1, 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "a7f95c7b-56d4-437b-a160-cabb8fd2b1e4", 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 60, 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6812", 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6815", 2024-09-16T13:17:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6818", 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6819", 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6822", 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6823", 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.147 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6820", 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6821", 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22272 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6815/22272", 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6819/22272", 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6823/22272", 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6821/22272", 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 2, 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c", 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:58.149 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 18, 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6800", 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:58.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6801", 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6802", 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6803", 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6806", 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6807", 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6804", 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6805", 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22273 2024-09-16T13:17:58.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6801/22273", 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6803/22273", 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6807/22273", 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6805/22273", 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 3, 2024-09-16T13:17:58.154 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "4898b5b5-b912-430c-adcb-c52d90226917", 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 61, 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6824", 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6825", 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6826", 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6827", 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:58.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6830", 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6831", 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:58.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6828", 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6829", 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22270 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.159 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6825/22270", 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.98:6827/22270", 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.98:6831/22270", 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.98:6829/22270", 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 4, 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "6c478be2-f606-41d8-967c-9a689f09a440", 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:58.160 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 61, 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6824", 2024-09-16T13:17:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6825", 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6826", 2024-09-16T13:17:58.162 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6827", 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6830", 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6831", 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6828", 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6829", 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22286 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6825/22286", 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6827/22286", 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6831/22286", 2024-09-16T13:17:58.165 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6829/22286", 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 5, 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "5001c828-b223-4d2c-b893-f9c56da73b83", 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:58.166 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 46, 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6808", 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6809", 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6810", 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6811", 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6814", 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6815", 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6812", 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6813", 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22285 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6809/22285", 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6811/22285", 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6815/22285", 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6813/22285", 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.171 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 6, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "6f7a9125-674c-4be2-93e1-b6296cd00f8e", 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 15, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 58, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6816", 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6817", 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6818", 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6819", 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6822", 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6823", 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6820", 2024-09-16T13:17:58.175 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6821", 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22315 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6817/22315", 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6819/22315", 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6823/22315", 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6821/22315", 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:58.176 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 7, 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "uuid": "bcdfe435-f24d-45a9-a629-7380ad2e575a", 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "up": 1, 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "in": 1, 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 1, 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_affinity": 1, 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_begin": 0, 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "last_clean_end": 0, 2024-09-16T13:17:58.177 INFO:teuthology.orchestra.run.smithi098.stdout: "up_from": 14, 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "up_thru": 58, 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "down_at": 0, 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "lost_at": 0, 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6800", 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6801", 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:58.178 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addrs": { 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6802", 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6803", 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.179 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addrs": { 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6806", 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6807", 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addrs": { 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6804", 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.151:6805", 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 22284 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.151:6801/22284", 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_addr": "172.21.15.151:6803/22284", 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_back_addr": "172.21.15.151:6807/22284", 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "heartbeat_front_addr": "172.21.15.151:6805/22284", 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "state": [ 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "exists", 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: "up" 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.182 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_xinfo": [ 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 0, 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.191311+0000", 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 1, 2024-09-16T13:17:58.183 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.563551+0000", 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 2, 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:58.184 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.493457+0000", 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 3, 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.599815+0000", 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 4, 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.634352+0000", 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 5, 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:58.186 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.262873+0000", 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 6, 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.508881+0000", 2024-09-16T13:17:58.187 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": 7, 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "down_stamp": "0.000000", 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_probability": 0, 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "laggy_interval": 0, 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "features": 4540138320759226367, 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "old_weight": 0, 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "last_purged_snaps_scrub": "2024-09-16T13:16:30.348276+0000", 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "dead_epoch": 0 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:58.188 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_upmap": [ 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "pgid": "3.1c", 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "osds": [ 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: 1 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_upmap_items": [], 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_temp": [], 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "primary_temp": [], 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklist": {}, 2024-09-16T13:17:58.189 INFO:teuthology.orchestra.run.smithi098.stdout: "range_blocklist": {}, 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "erasure_code_profiles": { 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "default": { 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-failure-domain": "osd", 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "k": "2", 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "m": "1", 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "plugin": "jerasure", 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "technique": "reed_sol_van" 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_snaps_queue": [], 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "new_removed_snaps": [], 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "new_purged_snaps": [], 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_node_flags": {}, 2024-09-16T13:17:58.190 INFO:teuthology.orchestra.run.smithi098.stdout: "device_class_flags": {}, 2024-09-16T13:17:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": { 2024-09-16T13:17:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode_enabled": false, 2024-09-16T13:17:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_bucket_count": 0, 2024-09-16T13:17:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "degraded_stretch_mode": 0, 2024-09-16T13:17:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "recovering_stretch_mode": 0, 2024-09-16T13:17:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode_bucket": 0 2024-09-16T13:17:58.191 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:58.191 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:58.193 INFO:tasks.thrashosds.thrasher:j is {'epoch': 72, 'fsid': '6ddcae89-6c18-4eb7-8712-1c53f3b07278', 'created': '2024-09-16T13:16:23.176689+0000', 'modified': '2024-09-16T13:17:58.057079+0000', 'last_up_change': '2024-09-16T13:16:33.765413+0000', 'last_in_change': '2024-09-16T13:17:42.548353+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:16:33.497561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:16:40.994538+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-16T13:16:50.759705+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 29, 'snap_epoch': 72, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': 'be8543b4-3d28-4b37-a4ce-d2a56a09aeaf', 'up': 1, 'in': 1, 'weight': 0.8320770263671875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6808', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6809', 'nonce': 22271}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6810', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6811', 'nonce': 22271}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6816', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6817', 'nonce': 22271}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6813', 'nonce': 22271}, {'type': 'v1', 'addr': '172.21.15.98:6814', 'nonce': 22271}]}, 'public_addr': '172.21.15.98:6809/22271', 'cluster_addr': '172.21.15.98:6811/22271', 'heartbeat_back_addr': '172.21.15.98:6817/22271', 'heartbeat_front_addr': '172.21.15.98:6814/22271', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'a7f95c7b-56d4-437b-a160-cabb8fd2b1e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 60, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6812', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6815', 'nonce': 22272}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6818', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6819', 'nonce': 22272}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6822', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6823', 'nonce': 22272}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6820', 'nonce': 22272}, {'type': 'v1', 'addr': '172.21.15.98:6821', 'nonce': 22272}]}, 'public_addr': '172.21.15.98:6815/22272', 'cluster_addr': '172.21.15.98:6819/22272', 'heartbeat_back_addr': '172.21.15.98:6823/22272', 'heartbeat_front_addr': '172.21.15.98:6821/22272', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6800', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6801', 'nonce': 22273}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6802', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6803', 'nonce': 22273}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6806', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6807', 'nonce': 22273}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6804', 'nonce': 22273}, {'type': 'v1', 'addr': '172.21.15.98:6805', 'nonce': 22273}]}, 'public_addr': '172.21.15.98:6801/22273', 'cluster_addr': '172.21.15.98:6803/22273', 'heartbeat_back_addr': '172.21.15.98:6807/22273', 'heartbeat_front_addr': '172.21.15.98:6805/22273', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4898b5b5-b912-430c-adcb-c52d90226917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6824', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6825', 'nonce': 22270}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6826', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6827', 'nonce': 22270}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6830', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6831', 'nonce': 22270}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6828', 'nonce': 22270}, {'type': 'v1', 'addr': '172.21.15.98:6829', 'nonce': 22270}]}, 'public_addr': '172.21.15.98:6825/22270', 'cluster_addr': '172.21.15.98:6827/22270', 'heartbeat_back_addr': '172.21.15.98:6831/22270', 'heartbeat_front_addr': '172.21.15.98:6829/22270', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6c478be2-f606-41d8-967c-9a689f09a440', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 22286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 22286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 22286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 22286}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 22286}]}, 'public_addr': '172.21.15.151:6825/22286', 'cluster_addr': '172.21.15.151:6827/22286', 'heartbeat_back_addr': '172.21.15.151:6831/22286', 'heartbeat_front_addr': '172.21.15.151:6829/22286', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5001c828-b223-4d2c-b893-f9c56da73b83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 22285}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 22285}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 22285}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 22285}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 22285}]}, 'public_addr': '172.21.15.151:6809/22285', 'cluster_addr': '172.21.15.151:6811/22285', 'heartbeat_back_addr': '172.21.15.151:6815/22285', 'heartbeat_front_addr': '172.21.15.151:6813/22285', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '6f7a9125-674c-4be2-93e1-b6296cd00f8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 22315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 22315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 22315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 22315}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 22315}]}, 'public_addr': '172.21.15.151:6817/22315', 'cluster_addr': '172.21.15.151:6819/22315', 'heartbeat_back_addr': '172.21.15.151:6823/22315', 'heartbeat_front_addr': '172.21.15.151:6821/22315', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'bcdfe435-f24d-45a9-a629-7380ad2e575a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6800', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6801', 'nonce': 22284}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 22284}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 22284}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 22284}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 22284}]}, 'public_addr': '172.21.15.151:6801/22284', 'cluster_addr': '172.21.15.151:6803/22284', 'heartbeat_back_addr': '172.21.15.151:6807/22284', 'heartbeat_front_addr': '172.21.15.151:6805/22284', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.191311+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.563551+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.493457+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.599815+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.634352+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.262873+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.508881+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T13:16:30.348276+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.1c', 'osds': [6, 1]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T13:17:58.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:17:58.193 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:58.184+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:58.211 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:58.212 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:58.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:17:58.293 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:58.284+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:58.304 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone25 from 0x9e972e0 (overlap 0x12da2df) to 0x10f78fb 2024-09-16T13:17:58.329 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:58.329 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:58.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:17:58.394 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:58.387+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:58.436 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:58.436 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:58.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:17:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:17:58.499 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:58.491+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:58.500 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:17:58.512 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-16T13:17:58.071210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662514128,"num_objects":938,"num_object_clones":240,"num_object_copies":1876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":938,"num_whiteouts":294,"num_read":46424,"num_read_kb":208237,"num_write":10561,"num_write_kb":106457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":22514126,"num_keys_recovered":124,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8400,"ondisk_log_size":8400,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2474876,"kb_used_data":29432,"kb_used_omap":13,"kb_used_meta":2445298,"kb_avail":752499844,"statfs":{"total":773094113280,"available":770559840256,"internally_reserved":0,"allocated":30138368,"data_stored":131976515,"data_compressed":10137496,"data_compressed_allocated":11321344,"data_compressed_original":118445713,"omap_allocated":13632,"internal_metadata":2503985856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":59819967,"num_objects":52,"num_object_clones":26,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":20,"num_read":1298,"num_read_kb":6564,"num_write":318,"num_write_kb":3803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.044357"},"pg_stats":[{"pgid":"3.1f","version":"67'223","reported_seq":891,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.433837+0000","last_change":"2024-09-16T13:17:02.968041+0000","last_active":"2024-09-16T13:17:53.433837+0000","last_peered":"2024-09-16T13:17:53.433837+0000","last_clean":"2024-09-16T13:17:53.433837+0000","last_became_active":"2024-09-16T13:17:02.950173+0000","last_became_peered":"2024-09-16T13:17:02.950173+0000","last_unstale":"2024-09-16T13:17:53.433837+0000","last_undegraded":"2024-09-16T13:17:53.433837+0000","last_fullsized":"2024-09-16T13:17:53.433837+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:04:46.146416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054641499999999996,"stat_sum":{"num_bytes":45488952,"num_objects":25,"num_object_clones":5,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":7,"num_read":1448,"num_read_kb":6410,"num_write":242,"num_write_kb":2005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"67'332","reported_seq":1682,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.554421+0000","last_change":"2024-09-16T13:16:54.298169+0000","last_active":"2024-09-16T13:17:53.554421+0000","last_peered":"2024-09-16T13:17:53.554421+0000","last_clean":"2024-09-16T13:17:53.554421+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:17:53.554421+0000","last_undegraded":"2024-09-16T13:17:53.554421+0000","last_fullsized":"2024-09-16T13:17:53.554421+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:50.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001135177,"stat_sum":{"num_bytes":5232130,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":3562,"num_read_kb":4315,"num_write":315,"num_write_kb":1915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"67'241","reported_seq":816,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.354688+0000","last_change":"2024-09-16T13:16:54.293554+0000","last_active":"2024-09-16T13:17:53.354688+0000","last_peered":"2024-09-16T13:17:53.354688+0000","last_clean":"2024-09-16T13:17:53.354688+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:17:53.354688+0000","last_undegraded":"2024-09-16T13:17:53.354688+0000","last_fullsized":"2024-09-16T13:17:53.354688+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:14.387827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001293439,"stat_sum":{"num_bytes":4386566,"num_objects":25,"num_object_clones":5,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":7,"num_read":1663,"num_read_kb":1750,"num_write":292,"num_write_kb":1320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"67'176","reported_seq":398,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.551989+0000","last_change":"2024-09-16T13:17:10.103002+0000","last_active":"2024-09-16T13:17:53.551989+0000","last_peered":"2024-09-16T13:17:53.551989+0000","last_clean":"2024-09-16T13:17:53.551989+0000","last_became_active":"2024-09-16T13:17:10.061217+0000","last_became_peered":"2024-09-16T13:17:10.061217+0000","last_unstale":"2024-09-16T13:17:53.551989+0000","last_undegraded":"2024-09-16T13:17:53.551989+0000","last_fullsized":"2024-09-16T13:17:53.551989+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:10.571278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010706929999999999,"stat_sum":{"num_bytes":16227415,"num_objects":30,"num_object_clones":8,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":8,"num_read":310,"num_read_kb":844,"num_write":171,"num_write_kb":1365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"66'489","reported_seq":957,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.023616+0000","last_change":"2024-09-16T13:17:02.955334+0000","last_active":"2024-09-16T13:17:53.023616+0000","last_peered":"2024-09-16T13:17:53.023616+0000","last_clean":"2024-09-16T13:17:53.023616+0000","last_became_active":"2024-09-16T13:17:02.948727+0000","last_became_peered":"2024-09-16T13:17:02.948727+0000","last_unstale":"2024-09-16T13:17:53.023616+0000","last_undegraded":"2024-09-16T13:17:53.023616+0000","last_fullsized":"2024-09-16T13:17:53.023616+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:17:17.238723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011449159999999999,"stat_sum":{"num_bytes":35247862,"num_objects":38,"num_object_clones":12,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":15,"num_read":2347,"num_read_kb":5919,"num_write":761,"num_write_kb":3784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"68'432","reported_seq":1690,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:17:54.447908+0000","last_change":"2024-09-16T13:16:54.304914+0000","last_active":"2024-09-16T13:17:54.447908+0000","last_peered":"2024-09-16T13:17:54.447908+0000","last_clean":"2024-09-16T13:17:54.447908+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:17:54.447908+0000","last_undegraded":"2024-09-16T13:17:54.447908+0000","last_fullsized":"2024-09-16T13:17:54.447908+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:31.071219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017517260000000001,"stat_sum":{"num_bytes":32506625,"num_objects":31,"num_object_clones":9,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":11,"num_read":2687,"num_read_kb":26111,"num_write":618,"num_write_kb":9983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'169","reported_seq":790,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.319522+0000","last_change":"2024-09-16T13:16:54.295771+0000","last_active":"2024-09-16T13:17:53.319522+0000","last_peered":"2024-09-16T13:17:53.319522+0000","last_clean":"2024-09-16T13:17:53.319522+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:17:53.319522+0000","last_undegraded":"2024-09-16T13:17:53.319522+0000","last_fullsized":"2024-09-16T13:17:53.319522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:42:30.327300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012778399999999999,"stat_sum":{"num_bytes":11211907,"num_objects":33,"num_object_clones":8,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":10,"num_read":1196,"num_read_kb":3671,"num_write":169,"num_write_kb":2052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"67'95","reported_seq":278,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.552407+0000","last_change":"2024-09-16T13:17:02.963249+0000","last_active":"2024-09-16T13:17:53.552407+0000","last_peered":"2024-09-16T13:17:53.552407+0000","last_clean":"2024-09-16T13:17:53.552407+0000","last_became_active":"2024-09-16T13:17:02.962736+0000","last_became_peered":"2024-09-16T13:17:02.962736+0000","last_unstale":"2024-09-16T13:17:53.552407+0000","last_undegraded":"2024-09-16T13:17:53.552407+0000","last_fullsized":"2024-09-16T13:17:53.552407+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:27:05.338093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012285180000000001,"stat_sum":{"num_bytes":10234967,"num_objects":23,"num_object_clones":7,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":157,"num_read_kb":2037,"num_write":99,"num_write_kb":1587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"66'358","reported_seq":1488,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.148989+0000","last_change":"2024-09-16T13:17:45.977171+0000","last_active":"2024-09-16T13:17:53.148989+0000","last_peered":"2024-09-16T13:17:53.148989+0000","last_clean":"2024-09-16T13:17:53.148989+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:17:53.148989+0000","last_undegraded":"2024-09-16T13:17:53.148989+0000","last_fullsized":"2024-09-16T13:17:53.148989+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:08:57.225261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087549000000000004,"stat_sum":{"num_bytes":20531000,"num_objects":39,"num_object_clones":11,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":12,"num_read":2933,"num_read_kb":5115,"num_write":436,"num_write_kb":1959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"67'117","reported_seq":541,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.463798+0000","last_change":"2024-09-16T13:16:54.292979+0000","last_active":"2024-09-16T13:17:53.463798+0000","last_peered":"2024-09-16T13:17:53.463798+0000","last_clean":"2024-09-16T13:17:53.463798+0000","last_became_active":"2024-09-16T13:16:52.296320+0000","last_became_peered":"2024-09-16T13:16:52.296320+0000","last_unstale":"2024-09-16T13:17:53.463798+0000","last_undegraded":"2024-09-16T13:17:53.463798+0000","last_fullsized":"2024-09-16T13:17:53.463798+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:42:35.683007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000822955,"stat_sum":{"num_bytes":5853905,"num_objects":20,"num_object_clones":5,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":7,"num_read":465,"num_read_kb":6523,"num_write":111,"num_write_kb":1826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"67'145","reported_seq":490,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.401935+0000","last_change":"2024-09-16T13:16:54.624096+0000","last_active":"2024-09-16T13:17:53.401935+0000","last_peered":"2024-09-16T13:17:53.401935+0000","last_clean":"2024-09-16T13:17:53.401935+0000","last_became_active":"2024-09-16T13:16:52.293972+0000","last_became_peered":"2024-09-16T13:16:52.293972+0000","last_unstale":"2024-09-16T13:17:53.401935+0000","last_undegraded":"2024-09-16T13:17:53.401935+0000","last_fullsized":"2024-09-16T13:17:53.401935+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:36.145926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001372786,"stat_sum":{"num_bytes":11098956,"num_objects":28,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":13,"num_read":322,"num_read_kb":7464,"num_write":128,"num_write_kb":1169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"66'129","reported_seq":257,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.023440+0000","last_change":"2024-09-16T13:16:54.305409+0000","last_active":"2024-09-16T13:17:53.023440+0000","last_peered":"2024-09-16T13:17:53.023440+0000","last_clean":"2024-09-16T13:17:53.023440+0000","last_became_active":"2024-09-16T13:16:52.297145+0000","last_became_peered":"2024-09-16T13:16:52.297145+0000","last_unstale":"2024-09-16T13:17:53.023440+0000","last_undegraded":"2024-09-16T13:17:53.023440+0000","last_fullsized":"2024-09-16T13:17:53.023440+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:04.575154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001794908,"stat_sum":{"num_bytes":23133463,"num_objects":20,"num_object_clones":5,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":7,"num_read":113,"num_read_kb":1602,"num_write":122,"num_write_kb":1610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.034985+0000","last_change":"2024-09-16T13:16:45.344339+0000","last_active":"2024-09-16T13:17:53.034985+0000","last_peered":"2024-09-16T13:17:53.034985+0000","last_clean":"2024-09-16T13:17:53.034985+0000","last_became_active":"2024-09-16T13:16:42.825708+0000","last_became_peered":"2024-09-16T13:16:42.825708+0000","last_unstale":"2024-09-16T13:17:53.034985+0000","last_undegraded":"2024-09-16T13:17:53.034985+0000","last_fullsized":"2024-09-16T13:17:53.034985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:23.678862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"66'485","reported_seq":815,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.490486+0000","last_change":"2024-09-16T13:17:45.979289+0000","last_active":"2024-09-16T13:17:52.490486+0000","last_peered":"2024-09-16T13:17:52.490486+0000","last_clean":"2024-09-16T13:17:52.490486+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:17:52.490486+0000","last_undegraded":"2024-09-16T13:17:52.490486+0000","last_fullsized":"2024-09-16T13:17:52.490486+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:52:05.144800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043376500000000002,"stat_sum":{"num_bytes":6870774,"num_objects":26,"num_object_clones":6,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":10,"num_read":2180,"num_read_kb":3208,"num_write":774,"num_write_kb":1572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":62,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.100158+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:17:53.100158+0000","last_peered":"2024-09-16T13:17:53.100158+0000","last_clean":"2024-09-16T13:17:53.100158+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:17:53.100158+0000","last_undegraded":"2024-09-16T13:17:53.100158+0000","last_fullsized":"2024-09-16T13:17:53.100158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"67'128","reported_seq":427,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.553672+0000","last_change":"2024-09-16T13:16:54.298553+0000","last_active":"2024-09-16T13:17:53.553672+0000","last_peered":"2024-09-16T13:17:53.553672+0000","last_clean":"2024-09-16T13:17:53.553672+0000","last_became_active":"2024-09-16T13:16:52.294439+0000","last_became_peered":"2024-09-16T13:16:52.294439+0000","last_unstale":"2024-09-16T13:17:53.553672+0000","last_undegraded":"2024-09-16T13:17:53.553672+0000","last_fullsized":"2024-09-16T13:17:53.553672+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:59:42.121642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013329209999999999,"stat_sum":{"num_bytes":24742957,"num_objects":29,"num_object_clones":10,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":9,"num_read":303,"num_read_kb":3949,"num_write":134,"num_write_kb":2102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.027261+0000","last_change":"2024-09-16T13:17:42.960494+0000","last_active":"2024-09-16T13:17:52.027261+0000","last_peered":"2024-09-16T13:17:52.027261+0000","last_clean":"2024-09-16T13:17:52.027261+0000","last_became_active":"2024-09-16T13:17:42.960104+0000","last_became_peered":"2024-09-16T13:17:42.960104+0000","last_unstale":"2024-09-16T13:17:52.027261+0000","last_undegraded":"2024-09-16T13:17:52.027261+0000","last_fullsized":"2024-09-16T13:17:52.027261+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:26:27.307046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"67'130","reported_seq":648,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.568445+0000","last_change":"2024-09-16T13:16:54.295888+0000","last_active":"2024-09-16T13:17:53.568445+0000","last_peered":"2024-09-16T13:17:53.568445+0000","last_clean":"2024-09-16T13:17:53.568445+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:17:53.568445+0000","last_undegraded":"2024-09-16T13:17:53.568445+0000","last_fullsized":"2024-09-16T13:17:53.568445+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:30.996264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084973399999999999,"stat_sum":{"num_bytes":47706677,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":7,"num_read":493,"num_read_kb":8427,"num_write":127,"num_write_kb":2995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":80,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.100028+0000","last_change":"2024-09-16T13:17:42.960492+0000","last_active":"2024-09-16T13:17:53.100028+0000","last_peered":"2024-09-16T13:17:53.100028+0000","last_clean":"2024-09-16T13:17:53.100028+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:17:53.100028+0000","last_undegraded":"2024-09-16T13:17:53.100028+0000","last_fullsized":"2024-09-16T13:17:53.100028+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:35:45.152557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'264","reported_seq":903,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.439832+0000","last_change":"2024-09-16T13:16:54.293894+0000","last_active":"2024-09-16T13:17:53.439832+0000","last_peered":"2024-09-16T13:17:53.439832+0000","last_clean":"2024-09-16T13:17:53.439832+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:17:53.439832+0000","last_undegraded":"2024-09-16T13:17:53.439832+0000","last_fullsized":"2024-09-16T13:17:53.439832+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:34.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015391249999999999,"stat_sum":{"num_bytes":29872198,"num_objects":41,"num_object_clones":8,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":13,"num_read":1506,"num_read_kb":7351,"num_write":308,"num_write_kb":1908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"67'180","reported_seq":868,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.911779+0000","last_change":"2024-09-16T13:17:45.975430+0000","last_active":"2024-09-16T13:17:53.911779+0000","last_peered":"2024-09-16T13:17:53.911779+0000","last_clean":"2024-09-16T13:17:53.911779+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:17:53.911779+0000","last_undegraded":"2024-09-16T13:17:53.911779+0000","last_fullsized":"2024-09-16T13:17:53.911779+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:44:16.402879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001425195,"stat_sum":{"num_bytes":4913737,"num_objects":27,"num_object_clones":9,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":11,"num_read":1262,"num_read_kb":2098,"num_write":177,"num_write_kb":1487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":63,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.110392+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:17:52.110392+0000","last_peered":"2024-09-16T13:17:52.110392+0000","last_clean":"2024-09-16T13:17:52.110392+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:17:52.110392+0000","last_undegraded":"2024-09-16T13:17:52.110392+0000","last_fullsized":"2024-09-16T13:17:52.110392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"68'174","reported_seq":505,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:17:54.241685+0000","last_change":"2024-09-16T13:16:54.294518+0000","last_active":"2024-09-16T13:17:54.241685+0000","last_peered":"2024-09-16T13:17:54.241685+0000","last_clean":"2024-09-16T13:17:54.241685+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:17:54.241685+0000","last_undegraded":"2024-09-16T13:17:54.241685+0000","last_fullsized":"2024-09-16T13:17:54.241685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:29.090161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046065400000000001,"stat_sum":{"num_bytes":23482008,"num_objects":28,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":10,"num_read":882,"num_read_kb":1316,"num_write":178,"num_write_kb":1322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.150508+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:17:52.150508+0000","last_peered":"2024-09-16T13:17:52.150508+0000","last_clean":"2024-09-16T13:17:52.150508+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:17:52.150508+0000","last_undegraded":"2024-09-16T13:17:52.150508+0000","last_fullsized":"2024-09-16T13:17:52.150508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"67'257","reported_seq":712,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.390076+0000","last_change":"2024-09-16T13:17:45.976543+0000","last_active":"2024-09-16T13:17:53.390076+0000","last_peered":"2024-09-16T13:17:53.390076+0000","last_clean":"2024-09-16T13:17:53.390076+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:17:53.390076+0000","last_undegraded":"2024-09-16T13:17:53.390076+0000","last_fullsized":"2024-09-16T13:17:53.390076+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:03.357310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067097399999999996,"stat_sum":{"num_bytes":16862063,"num_objects":27,"num_object_clones":9,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":10,"num_read":955,"num_read_kb":22033,"num_write":374,"num_write_kb":12323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.026743+0000","last_change":"2024-09-16T13:16:45.344856+0000","last_active":"2024-09-16T13:17:52.026743+0000","last_peered":"2024-09-16T13:17:52.026743+0000","last_clean":"2024-09-16T13:17:52.026743+0000","last_became_active":"2024-09-16T13:16:42.825917+0000","last_became_peered":"2024-09-16T13:16:42.825917+0000","last_unstale":"2024-09-16T13:17:52.026743+0000","last_undegraded":"2024-09-16T13:17:52.026743+0000","last_fullsized":"2024-09-16T13:17:52.026743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:30.133715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082018499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"67'277","reported_seq":1303,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.885572+0000","last_change":"2024-09-16T13:16:54.298588+0000","last_active":"2024-09-16T13:17:53.885572+0000","last_peered":"2024-09-16T13:17:53.885572+0000","last_clean":"2024-09-16T13:17:53.885572+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:17:53.885572+0000","last_undegraded":"2024-09-16T13:17:53.885572+0000","last_fullsized":"2024-09-16T13:17:53.885572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:45.154984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001266124,"stat_sum":{"num_bytes":22090205,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":10,"num_read":2535,"num_read_kb":5382,"num_write":268,"num_write_kb":3706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":62,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.110472+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:17:52.110472+0000","last_peered":"2024-09-16T13:17:52.110472+0000","last_clean":"2024-09-16T13:17:52.110472+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:17:52.110472+0000","last_undegraded":"2024-09-16T13:17:52.110472+0000","last_fullsized":"2024-09-16T13:17:52.110472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":179,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.026784+0000","last_change":"2024-09-16T13:17:42.966879+0000","last_active":"2024-09-16T13:17:52.026784+0000","last_peered":"2024-09-16T13:17:52.026784+0000","last_clean":"2024-09-16T13:17:52.026784+0000","last_became_active":"2024-09-16T13:17:42.960921+0000","last_became_peered":"2024-09-16T13:17:42.960921+0000","last_unstale":"2024-09-16T13:17:52.026784+0000","last_undegraded":"2024-09-16T13:17:52.026784+0000","last_fullsized":"2024-09-16T13:17:52.026784+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:07.660612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"68'454","reported_seq":671,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:17:54.425967+0000","last_change":"2024-09-16T13:16:54.297765+0000","last_active":"2024-09-16T13:17:54.425967+0000","last_peered":"2024-09-16T13:17:54.425967+0000","last_clean":"2024-09-16T13:17:54.425967+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:17:54.425967+0000","last_undegraded":"2024-09-16T13:17:54.425967+0000","last_fullsized":"2024-09-16T13:17:54.425967+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:59.515115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012012500000000001,"stat_sum":{"num_bytes":25722333,"num_objects":27,"num_object_clones":6,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":4,"num_read":1710,"num_read_kb":16037,"num_write":666,"num_write_kb":9656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"68'232","reported_seq":982,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:17:54.318528+0000","last_change":"2024-09-16T13:16:54.303746+0000","last_active":"2024-09-16T13:17:54.318528+0000","last_peered":"2024-09-16T13:17:54.318528+0000","last_clean":"2024-09-16T13:17:54.318528+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:17:54.318528+0000","last_undegraded":"2024-09-16T13:17:54.318528+0000","last_fullsized":"2024-09-16T13:17:54.318528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:49:10.301550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065283599999999995,"stat_sum":{"num_bytes":53808622,"num_objects":41,"num_object_clones":13,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":15,"num_read":1453,"num_read_kb":15910,"num_write":230,"num_write_kb":8949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'581","reported_seq":1030,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:17:54.377399+0000","last_change":"2024-09-16T13:16:54.328489+0000","last_active":"2024-09-16T13:17:54.377399+0000","last_peered":"2024-09-16T13:17:54.377399+0000","last_clean":"2024-09-16T13:17:54.377399+0000","last_became_active":"2024-09-16T13:16:52.299458+0000","last_became_peered":"2024-09-16T13:16:52.299458+0000","last_unstale":"2024-09-16T13:17:54.377399+0000","last_undegraded":"2024-09-16T13:17:54.377399+0000","last_fullsized":"2024-09-16T13:17:54.377399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:34.071306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069184499999999996,"stat_sum":{"num_bytes":42249375,"num_objects":34,"num_object_clones":9,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":9,"num_read":2635,"num_read_kb":5859,"num_write":880,"num_write_kb":2038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"67'892","reported_seq":1952,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.749109+0000","last_change":"2024-09-16T13:17:45.975159+0000","last_active":"2024-09-16T13:17:53.749109+0000","last_peered":"2024-09-16T13:17:53.749109+0000","last_clean":"2024-09-16T13:17:53.749109+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:17:53.749109+0000","last_undegraded":"2024-09-16T13:17:53.749109+0000","last_fullsized":"2024-09-16T13:17:53.749109+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":892,"log_dups_size":0,"ondisk_log_size":892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:20.684083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00121396,"stat_sum":{"num_bytes":4621456,"num_objects":36,"num_object_clones":6,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":9,"num_read":5723,"num_read_kb":5909,"num_write":1448,"num_write_kb":2406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"66'154","reported_seq":726,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.490702+0000","last_change":"2024-09-16T13:16:54.292717+0000","last_active":"2024-09-16T13:17:52.490702+0000","last_peered":"2024-09-16T13:17:52.490702+0000","last_clean":"2024-09-16T13:17:52.490702+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:17:52.490702+0000","last_undegraded":"2024-09-16T13:17:52.490702+0000","last_fullsized":"2024-09-16T13:17:52.490702+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:39.937083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065643500000000003,"stat_sum":{"num_bytes":12581155,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":5,"num_read":1002,"num_read_kb":9417,"num_write":152,"num_write_kb":3218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"66'178","reported_seq":992,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.605566+0000","last_change":"2024-09-16T13:16:54.624013+0000","last_active":"2024-09-16T13:17:52.605566+0000","last_peered":"2024-09-16T13:17:52.605566+0000","last_clean":"2024-09-16T13:17:52.605566+0000","last_became_active":"2024-09-16T13:16:52.294924+0000","last_became_peered":"2024-09-16T13:16:52.294924+0000","last_unstale":"2024-09-16T13:17:52.605566+0000","last_undegraded":"2024-09-16T13:17:52.605566+0000","last_fullsized":"2024-09-16T13:17:52.605566+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:49:58.037805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012996780000000001,"stat_sum":{"num_bytes":38147056,"num_objects":34,"num_object_clones":6,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":11,"num_read":1085,"num_read_kb":2627,"num_write":164,"num_write_kb":1690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"67'125","reported_seq":299,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.449137+0000","last_change":"2024-09-16T13:16:54.623553+0000","last_active":"2024-09-16T13:17:53.449137+0000","last_peered":"2024-09-16T13:17:53.449137+0000","last_clean":"2024-09-16T13:17:53.449137+0000","last_became_active":"2024-09-16T13:16:52.299462+0000","last_became_peered":"2024-09-16T13:16:52.299462+0000","last_unstale":"2024-09-16T13:17:53.449137+0000","last_undegraded":"2024-09-16T13:17:53.449137+0000","last_fullsized":"2024-09-16T13:17:53.449137+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:23:35.582311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087311099999999998,"stat_sum":{"num_bytes":13083564,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":7,"num_read":217,"num_read_kb":1460,"num_write":124,"num_write_kb":1156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"67'164","reported_seq":627,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.082742+0000","last_change":"2024-09-16T13:16:54.624805+0000","last_active":"2024-09-16T13:17:53.082742+0000","last_peered":"2024-09-16T13:17:53.082742+0000","last_clean":"2024-09-16T13:17:53.082742+0000","last_became_active":"2024-09-16T13:16:52.299393+0000","last_became_peered":"2024-09-16T13:16:52.299393+0000","last_unstale":"2024-09-16T13:17:53.082742+0000","last_undegraded":"2024-09-16T13:17:53.082742+0000","last_fullsized":"2024-09-16T13:17:53.082742+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:20.663757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019518529999999999,"stat_sum":{"num_bytes":45996784,"num_objects":45,"num_object_clones":11,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":14,"num_read":594,"num_read_kb":10150,"num_write":173,"num_write_kb":4484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"68'126","reported_seq":643,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-16T13:17:54.398546+0000","last_change":"2024-09-16T13:17:43.131706+0000","last_active":"2024-09-16T13:17:54.398546+0000","last_peered":"2024-09-16T13:17:54.398546+0000","last_clean":"2024-09-16T13:17:54.398546+0000","last_became_active":"2024-09-16T13:17:42.961695+0000","last_became_peered":"2024-09-16T13:17:42.961695+0000","last_unstale":"2024-09-16T13:17:54.398546+0000","last_undegraded":"2024-09-16T13:17:54.398546+0000","last_fullsized":"2024-09-16T13:17:54.398546+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:35.675244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049957799999999998,"stat_sum":{"num_bytes":11511189,"num_objects":27,"num_object_clones":8,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":7,"num_read":335,"num_read_kb":6626,"num_write":143,"num_write_kb":5368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"66'143","reported_seq":369,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.605868+0000","last_change":"2024-09-16T13:16:54.625008+0000","last_active":"2024-09-16T13:17:52.605868+0000","last_peered":"2024-09-16T13:17:52.605868+0000","last_clean":"2024-09-16T13:17:52.605868+0000","last_became_active":"2024-09-16T13:16:52.300728+0000","last_became_peered":"2024-09-16T13:16:52.300728+0000","last_unstale":"2024-09-16T13:17:52.605868+0000","last_undegraded":"2024-09-16T13:17:52.605868+0000","last_fullsized":"2024-09-16T13:17:52.605868+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:56:53.877292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001790673,"stat_sum":{"num_bytes":7544624,"num_objects":29,"num_object_clones":8,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":11,"num_read":354,"num_read_kb":4091,"num_write":173,"num_write_kb":3995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"64'150","reported_seq":782,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T13:17:52.032458+0000","last_change":"2024-09-16T13:17:45.976332+0000","last_active":"2024-09-16T13:17:52.032458+0000","last_peered":"2024-09-16T13:17:52.032458+0000","last_clean":"2024-09-16T13:17:52.032458+0000","last_became_active":"2024-09-16T13:17:45.975609+0000","last_became_peered":"2024-09-16T13:17:45.975609+0000","last_unstale":"2024-09-16T13:17:52.032458+0000","last_undegraded":"2024-09-16T13:17:52.032458+0000","last_fullsized":"2024-09-16T13:17:52.032458+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:40:24.863359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049211100000000005,"stat_sum":{"num_bytes":4455617,"num_objects":10,"num_object_clones":2,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":1260,"num_read_kb":1528,"num_write":151,"num_write_kb":2242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":174069,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"67'325","reported_seq":1187,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-16T13:17:53.378774+0000","last_change":"2024-09-16T13:16:54.623401+0000","last_active":"2024-09-16T13:17:53.378774+0000","last_peered":"2024-09-16T13:17:53.378774+0000","last_clean":"2024-09-16T13:17:53.378774+0000","last_became_active":"2024-09-16T13:16:52.299393+0000","last_became_peered":"2024-09-16T13:16:52.299393+0000","last_unstale":"2024-09-16T13:17:53.378774+0000","last_undegraded":"2024-09-16T13:17:53.378774+0000","last_fullsized":"2024-09-16T13:17:53.378774+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:08:01.819612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079151600000000003,"stat_sum":{"num_bytes":4507599,"num_objects":34,"num_object_clones":8,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":10,"num_read":2671,"num_read_kb":3043,"num_write":350,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":661923741,"num_objects":934,"num_object_clones":240,"num_object_copies":1868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":934,"num_whiteouts":294,"num_read":46358,"num_read_kb":208182,"num_write":10458,"num_write_kb":104827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":21333390,"num_keys_recovered":124,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23883776,"data_stored":128917544,"data_compressed":10174519,"data_compressed_allocated":11259904,"data_compressed_original":118154897,"omap_allocated":0,"internal_metadata":0},"log_size":8325,"ondisk_log_size":8325,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509460,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307624,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064216,"statfs":{"total":96636764160,"available":96321757184,"internally_reserved":0,"allocated":1990656,"data_stored":5243107,"data_compressed":286912,"data_compressed_allocated":425984,"data_compressed_original":4122814,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.57599999999999996}]},{"osd":1,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":2.8809999999999998},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":2.9169999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":4.6859999999999999,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":0.80400000000000005}]},{"osd":3,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.7930000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.006,"5min":1.006,"15min":1.006},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":3.0209999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.075,"5min":1.075,"15min":1.075},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.88800000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0110000000000001,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":1.0109999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.107},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.0700000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303392,"kb_used_data":6008,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94068448,"statfs":{"total":96636764160,"available":96326090752,"internally_reserved":0,"allocated":6152192,"data_stored":35843832,"data_compressed":2448212,"data_compressed_allocated":2916352,"data_compressed_original":33390291,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":0.372}]},{"osd":1,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.50600000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.83299999999999996}]},{"osd":3,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.71199999999999997}]},{"osd":5,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.432},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.68300000000000005}]},{"osd":6,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":0.76600000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509460,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318616,"kb_used_data":4620,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94053224,"statfs":{"total":96636764160,"available":96310501376,"internally_reserved":0,"allocated":4730880,"data_stored":17958269,"data_compressed":1431672,"data_compressed_allocated":1585152,"data_compressed_original":15566909,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.65200000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":1.1419999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.81299999999999994}]},{"osd":4,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":0.996},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":1.1930000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":1.052},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.032}]},{"osd":6,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":1.087},{"interface":"front","average":{"1min":1.01,"5min":1.01,"15min":1.01},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.96399999999999997}]},{"osd":7,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.226},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.1719999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509460,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310152,"kb_used_data":4472,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061688,"statfs":{"total":96636764160,"available":96319168512,"internally_reserved":0,"allocated":4579328,"data_stored":24961104,"data_compressed":1745233,"data_compressed_allocated":2002944,"data_compressed_original":23106732,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.78800000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.499},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.67800000000000005}]},{"osd":4,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.81499999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.871},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":1}]},{"osd":6,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.569,"5min":1.569,"15min":1.569},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.93700000000000006}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.96599999999999997}]}]},{"osd":7,"up_from":14,"seq":60129542163,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309332,"kb_used_data":3656,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062508,"statfs":{"total":96636764160,"available":96320008192,"internally_reserved":0,"allocated":3743744,"data_stored":13195228,"data_compressed":1032994,"data_compressed_allocated":1056768,"data_compressed_original":11200449,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":1.02}]},{"osd":1,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":2.5419999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.91200000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.131},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.755}]},{"osd":3,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":3.5499999999999998,"5min":3.5499999999999998,"15min":3.5499999999999998},"last":1.155},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":1.1080000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.94199999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":1.0509999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":1.0700000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308888,"kb_used_data":3212,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062952,"statfs":{"total":96636764160,"available":96320462848,"internally_reserved":0,"allocated":3289088,"data_stored":7515392,"data_compressed":962680,"data_compressed_allocated":851968,"data_compressed_original":5792152,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:17:32 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":1.141}]},{"osd":1,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":1.042},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.84199999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:17:32 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.92200000000000004}]},{"osd":3,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":1.196},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.0129999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.71199999999999997}]},{"osd":6,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":1.105},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.88400000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.0840000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542162,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303888,"kb_used_data":2376,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94067952,"statfs":{"total":96636764160,"available":96325582848,"internally_reserved":0,"allocated":2433024,"data_stored":13241197,"data_compressed":1119425,"data_compressed_allocated":1052672,"data_compressed_original":12406784,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.46700000000000003}]},{"osd":1,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.91200000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":2.8290000000000002,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.70799999999999996}]},{"osd":4,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.754},{"interface":"front","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.004}]},{"osd":5,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.7930000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.226}]},{"osd":6,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":1.3440000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":0.93899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312984,"kb_used_data":3144,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058856,"statfs":{"total":96636764160,"available":96316268544,"internally_reserved":0,"allocated":3219456,"data_stored":14018386,"data_compressed":1110368,"data_compressed_allocated":1429504,"data_compressed_original":12859582,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.498},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":0.76500000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.80300000000000005}]},{"osd":4,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":1.0600000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":1.04}]},{"osd":6,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.0820000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.881,"5min":1.881,"15min":1.881},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.2290000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2408448,"data_stored":13371622,"data_compressed":1152752,"data_compressed_allocated":1388544,"data_compressed_original":12556478,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3825664,"data_stored":24635569,"data_compressed":1745233,"data_compressed_allocated":2002944,"data_compressed_original":23106732,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1671168,"data_stored":12941465,"data_compressed":1121554,"data_compressed_allocated":1056768,"data_compressed_original":12451840,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3952640,"data_stored":17615106,"data_compressed":1432072,"data_compressed_allocated":1589248,"data_compressed_original":15575101,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5300224,"data_stored":35406530,"data_compressed":2446515,"data_compressed_allocated":2912256,"data_compressed_original":33357523,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2523136,"data_stored":7214769,"data_compressed":964623,"data_compressed_allocated":856064,"data_compressed_original":5833112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":4895703,"data_compressed":286912,"data_compressed_allocated":425984,"data_compressed_original":4122814,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2985984,"data_stored":12836780,"data_compressed":1024858,"data_compressed_allocated":1028096,"data_compressed_original":11151297,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:17:58.515 INFO:tasks.thrashosds.thrasher:Setting 3.1f to [5, 3] 2024-09-16T13:17:58.515 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.1f', '5', '3'] 2024-09-16T13:17:58.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.1f 5 3 2024-09-16T13:17:58.549 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:58.549 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:58.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:17:58.597 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:17:58.588+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:17:58.649 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #24, image image_client.0-clone24 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent25 2024-09-16T13:17:58.664 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:58.664 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:58.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:17:58.698 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:58.691+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:58.782 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:58.783 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:58.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:17:58.833 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:17:58.827+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:17:58.901 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:58.892+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:58.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:17:58.909 INFO:teuthology.orchestra.run.smithi098.stdout:4120 punch from 0x31cbe34 to 0x31ce7d9, (0x29a5 bytes) 2024-09-16T13:17:58.919 INFO:teuthology.orchestra.run.smithi098.stdout:4121 trunc from 0x9e972e0 to 0xb30d48c 2024-09-16T13:17:58.928 INFO:teuthology.orchestra.run.smithi098.stdout:4122 write 0x326f1a0 thru 0x326fd29 (0xb8a bytes) 2024-09-16T13:17:58.933 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:58.945 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:17:58.945 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:17:58.946 INFO:teuthology.orchestra.run.smithi098.stdout:4124 writesame 0xcdfa854 thru 0xce00745 data_size 0x3(0x5ef2 bytes) 2024-09-16T13:17:58.946 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:58.946 INFO:teuthology.orchestra.run.smithi098.stdout:4127 punch from 0x9b7543f to 0x9b7e2da, (0x8e9b bytes) 2024-09-16T13:17:58.946 INFO:teuthology.orchestra.run.smithi098.stdout:4128 read 0x9377547 thru 0x937a06c (0x2b26 bytes) 2024-09-16T13:17:58.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:17:58.964 INFO:teuthology.orchestra.run.smithi098.stdout:4129 trunc from 0xce00746 to 0x2926ac3 2024-09-16T13:17:58.994 INFO:teuthology.orchestra.run.smithi098.stdout:4130 punch from 0x1e5a223 to 0x1e6914f, (0xef2c bytes) 2024-09-16T13:17:59.002 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:17:58.992+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:17:59.002 INFO:teuthology.orchestra.run.smithi098.stdout:4131 trunc from 0x2926ac3 to 0x76627a1 2024-09-16T13:17:59.013 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:59.013 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:59.013 INFO:teuthology.orchestra.run.smithi098.stdout:4134 read 0x64df3c1 thru 0x64e2bc7 (0x3807 bytes) 2024-09-16T13:17:59.013 INFO:teuthology.orchestra.run.smithi098.stdout:4135 read 0x6f325a4 thru 0x6f35f8b (0x39e8 bytes) 2024-09-16T13:17:59.014 INFO:teuthology.orchestra.run.smithi098.stdout:4136 punch from 0x4002f10 to 0x400f87a, (0xc96a bytes) 2024-09-16T13:17:59.016 INFO:teuthology.orchestra.run.smithi098.stdout:4138 trunc from 0x76627a1 to 0x5ffdf15 2024-09-16T13:17:59.027 INFO:teuthology.orchestra.run.smithi098.stdout:4139 punch from 0x51db944 to 0x51e5b6e, (0xa22a bytes) 2024-09-16T13:17:59.027 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:17:59.027 INFO:teuthology.orchestra.run.smithi098.stdout:4142 writesame 0x25a042e thru 0x25ad1ca data_size 0x1(0xcd9d bytes) 2024-09-16T13:17:59.046 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:17:59.065 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:59.066 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:59.066 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:59.066 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:59.074 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:17:59.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:17:59.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:17:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:set 3.1f pg_upmap mapping to [5,3] 2024-09-16T13:17:59.083 INFO:teuthology.orchestra.run.smithi098.stdout:4146 trunc from 0x5ffdf15 to 0x343a16c 2024-09-16T13:17:59.095 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:17:59.099 INFO:teuthology.orchestra.run.smithi098.stdout:4147 read 0xabdc00 thru 0xacccc6 (0xf0c7 bytes) 2024-09-16T13:17:59.102 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:59.092+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:59.114 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:17:59.114 INFO:teuthology.orchestra.run.smithi098.stdout:4148 trunc from 0x343a16c to 0xb86ddf7 2024-09-16T13:17:59.126 INFO:teuthology.orchestra.run.smithi098.stdout:4149 write 0xaad508f thru 0xaad8557 (0x34c9 bytes) 2024-09-16T13:17:59.133 INFO:teuthology.orchestra.run.smithi098.stdout:4150 read 0x925310e thru 0x92600ba (0xcfad bytes) 2024-09-16T13:17:59.133 INFO:teuthology.orchestra.run.smithi098.stdout:4151 read 0xa677331 thru 0xa68089f (0x956f bytes) 2024-09-16T13:17:59.133 INFO:teuthology.orchestra.run.smithi098.stdout:4152 punch from 0x661de51 to 0x662c7be, (0xe96d bytes) 2024-09-16T13:17:59.133 INFO:teuthology.orchestra.run.smithi098.stdout:4153 read 0x5ff6103 thru 0x6001d5f (0xbc5d bytes) 2024-09-16T13:17:59.134 INFO:teuthology.orchestra.run.smithi098.stdout:4156 trunc from 0xb86ddf7 to 0xbe11db5 2024-09-16T13:17:59.146 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:17:59.148 INFO:teuthology.orchestra.run.smithi098.stdout:4159 write 0x276b765 thru 0x2770873 (0x510f bytes) 2024-09-16T13:17:59.152 INFO:teuthology.orchestra.run.smithi098.stdout:4161 write 0x2719ebd thru 0x272120a (0x734e bytes) 2024-09-16T13:17:59.156 INFO:teuthology.orchestra.run.smithi098.stdout:4162 read 0x43694c5 thru 0x436e343 (0x4e7f bytes) 2024-09-16T13:17:59.157 INFO:teuthology.orchestra.run.smithi098.stdout:4163 read 0xdfe8b3 thru 0xdffcba (0x1408 bytes) 2024-09-16T13:17:59.158 INFO:teuthology.orchestra.run.smithi098.stdout:4165 clone 27 order 18 su 32768 sc 5 2024-09-16T13:17:59.179 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:17:59.185 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:59.185 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:59.185 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:59.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:17:59.199 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:17:59.203 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:59.195+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:59.222 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:17:59.224 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:17:59.224 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:17:59.224 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:17:59.224 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:17:59.224 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:17:59.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:17:59.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:17:59.225 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:17:59.292 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.292 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11500 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 356352~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.789894639000003, 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014537983000000001, 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11500 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.293 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496474+0000", 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.801e-06 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496480+0000", 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1140000000000002e-06 2024-09-16T13:17:59.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496484+0000", 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8770000000000003e-06 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507573+0000", 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011088473 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507624+0000", 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1591999999999997e-05 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511010+0000", 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033857380000000001 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.297 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11501 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 430080~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.789878272000003, 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014726846, 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11501 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:17:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496489+0000", 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3700000000000001e-07 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496491+0000", 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9570000000000001e-06 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496497+0000", 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2550000000000003e-06 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496501+0000", 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8639999999999998e-06 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511060+0000", 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014559462 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511125+0000", 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4399999999999993e-05 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511215+0000", 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0470999999999995e-05 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11530 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 339968~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.302 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.788944084000001, 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014141661, 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11530 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497423+0000", 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-07 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497434+0000", 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1195e-05 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497442+0000", 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.323e-06 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497446+0000", 2024-09-16T13:17:59.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1049999999999997e-06 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511275+0000", 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013828872000000001 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511389+0000", 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114493 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511564+0000", 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017508700000000001 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11533 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 413696~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.788915000000003, 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014296237999999999, 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11533 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.51e-07 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497458+0000", 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9139999999999996e-06 2024-09-16T13:17:59.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497464+0000", 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0469999999999999e-06 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497467+0000", 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5030000000000002e-06 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511593+0000", 2024-09-16T13:17:59.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014125692 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511669+0000", 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5950000000000003e-05 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511748+0000", 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8681000000000001e-05 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11551 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 335872~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.788858973000004, 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014415314, 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11551 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:17:59.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497510+0000", 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.012e-06 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497516+0000", 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7880000000000003e-06 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497524+0000", 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9049999999999997e-06 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511769+0000", 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245809 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511818+0000", 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8092e-05 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511923+0000", 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010532299999999999 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11553 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 413696~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.788836123999999, 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014561233999999999, 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.316 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11553 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497531+0000", 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1899999999999998e-07 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497533+0000", 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9149999999999999e-06 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497539+0000", 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.088e-06 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497550+0000", 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0834e-05 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511962+0000", 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411947 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512018+0000", 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6755999999999997e-05 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512092+0000", 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3275000000000006e-05 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11577 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 409600~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.786728879999998, 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013920979999999999, 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.320 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11577 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499638+0000", 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499640+0000", 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1430000000000001e-06 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499646+0000", 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1029999999999998e-06 2024-09-16T13:17:59.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.499650+0000", 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6370000000000001e-06 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513442+0000", 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013791805000000001 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513494+0000", 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2074999999999999e-05 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.513559+0000", 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4874000000000003e-05 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11637 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.785836449000001, 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013965546000000001, 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11637 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500531+0000", 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2699999999999999e-07 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500534+0000", 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.055e-06 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500542+0000", 2024-09-16T13:17:59.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6389999999999998e-06 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500553+0000", 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0974e-05 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514249+0000", 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013696086 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514308+0000", 2024-09-16T13:17:59.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8269000000000001e-05 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514496+0000", 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018799600000000001 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11638 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.785806966999999, 2024-09-16T13:17:59.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014058684, 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11638 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500563+0000", 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0790000000000001e-06 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500580+0000", 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6472000000000001e-05 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500586+0000", 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2199999999999997e-06 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514522+0000", 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013935774 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514564+0000", 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2549000000000002e-05 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514618+0000", 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3942e-05 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11642 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:59.332 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.785722133999997, 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014093902, 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11642 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500645+0000", 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5200000000000002e-07 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500656+0000", 2024-09-16T13:17:59.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1372999999999999e-05 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500664+0000", 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9019999999999999e-06 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500677+0000", 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2954e-05 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514649+0000", 2024-09-16T13:17:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013971417 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514692+0000", 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3154000000000001e-05 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514738+0000", 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6650000000000002e-05 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11646 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.785683872, 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245793, 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11646 2024-09-16T13:17:59.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500685+0000", 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2019999999999998e-06 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500698+0000", 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.279e-05 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500702+0000", 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7629999999999998e-06 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514772+0000", 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014070384999999999 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514865+0000", 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2980999999999996e-05 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514928+0000", 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3114000000000006e-05 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11648 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.785660741999997, 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411604999999999, 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.341 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11648 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9900000000000001e-07 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500713+0000", 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.793e-06 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500719+0000", 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:17:59.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500722+0000", 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3809999999999999e-06 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514973+0000", 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014250924 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515031+0000", 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7864e-05 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515117+0000", 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6697e-05 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11654 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.785293908, 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014438998, 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.345 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11654 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8699999999999995e-07 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501075+0000", 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501082+0000", 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.844e-06 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501086+0000", 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7749999999999999e-06 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515152+0000", 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014065891000000001 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515232+0000", 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0461000000000001e-05 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515512+0000", 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027934599999999998 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11675 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.784831314000002, 2024-09-16T13:17:59.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014239586, 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11675 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501536+0000", 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501539+0000", 2024-09-16T13:17:59.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3529999999999999e-06 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501546+0000", 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.109e-06 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501551+0000", 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5109999999999997e-06 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515586+0000", 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014035334999999999 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515689+0000", 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010244200000000001 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515775+0000", 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6150999999999996e-05 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11679 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.784781389999999, 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014462922, 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11679 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501586+0000", 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501588+0000", 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3760000000000002e-06 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501594+0000", 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9950000000000002e-06 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501598+0000", 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7239999999999998e-06 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515840+0000", 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014242546999999999 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515949+0000", 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108591 2024-09-16T13:17:59.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516048+0000", 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9123000000000006e-05 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11686 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.784764817999999, 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014648357000000001, 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11686 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501604+0000", 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.846e-06 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501609+0000", 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3630000000000002e-06 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501613+0000", 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3749999999999999e-06 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516077+0000", 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014464351 2024-09-16T13:17:59.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516173+0000", 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5742000000000006e-05 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516250+0000", 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7312999999999997e-05 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11689 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.784750254000002, 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014770922000000001, 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11689 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501617+0000", 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:17:59.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501618+0000", 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6780000000000001e-06 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501624+0000", 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3750000000000002e-06 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501627+0000", 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5149999999999998e-06 2024-09-16T13:17:59.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516279+0000", 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014651928999999999 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516336+0000", 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6724000000000001e-05 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516387+0000", 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1334e-05 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11778 3.1 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.767416126000001, 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022495198000000001, 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.367 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11778 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518952+0000", 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.711e-06 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522261+0000", 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033087429999999998 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522296+0000", 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5335e-05 2024-09-16T13:17:59.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522306+0000", 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0074999999999999e-05 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522379+0000", 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2554999999999999e-05 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522572+0000", 2024-09-16T13:17:59.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019323100000000001 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.523266+0000", 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069379000000000001 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.534086+0000", 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010820112999999999 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:59.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541312+0000", 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072264440000000003 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541394+0000", 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1710000000000002e-05 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541446+0000", 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1490999999999999e-05 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:25474 3.6 3:6380c99b:::rbd_object_map.112de78841b1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:49.832331+0000", 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.4540353509999999, 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014819143999999999, 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 25474 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832331+0000", 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832331+0000", 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832332+0000", 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.15e-07 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832334+0000", 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.148e-06 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832337+0000", 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832340+0000", 2024-09-16T13:17:59.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4760000000000001e-06 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.842963+0000", 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010623886000000001 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.843120+0000", 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015636400000000001 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:59.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.843657+0000", 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000536887 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846702+0000", 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030451300000000001 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.847054+0000", 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035228400000000002 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.847116+0000", 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1977000000000006e-05 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.847150+0000", 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4257000000000001e-05 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:25567 3.6 3:6380c99b:::rbd_object_map.112de78841b1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:17:59.378 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.3036548420000003, 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014520869, 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 25567 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7100000000000002e-07 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982715+0000", 2024-09-16T13:17:59.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0639999999999998e-06 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982736+0000", 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0421999999999999e-05 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982743+0000", 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2509999999999997e-06 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.990894+0000", 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0081508239999999992 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.991014+0000", 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011978599999999999 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.991456+0000", 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044261099999999998 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.992802+0000", 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001345398 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.997165+0000", 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043631399999999997 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.997200+0000", 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5474e-05 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.997233+0000", 2024-09-16T13:17:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2327999999999999e-05 2024-09-16T13:17:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.384 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.384 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:59.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:17:59.386 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:17:59.376+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:17:59.482 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:59.484 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:59.484 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:59.484 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:59.486 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:59.476+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:59.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:17:59.588 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:17:59.583+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:17:59.643 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:59.643 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:59.643 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:17:59.643 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:17:59.643 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:59.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:17:59.689 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:59.683+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:59.757 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:59.757 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:17:59.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:17:59.757 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:17:59.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.757 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11223 3.8 e40/23 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head v 40'368, mlcod=39'365)", 2024-09-16T13:17:59.757 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:59.757 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.284702635000002, 2024-09-16T13:17:59.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01312405, 2024-09-16T13:17:59.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466704+0000", 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.792e-06 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471144+0000", 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044394830000000001 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471180+0000", 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5884999999999999e-05 2024-09-16T13:17:59.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471194+0000", 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4258e-05 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471273+0000", 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9159999999999994e-05 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471305+0000", 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2236000000000001e-05 2024-09-16T13:17:59.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479753+0000", 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0084474049999999998 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479826+0000", 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3831000000000005e-05 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11776 3.1a e40/23 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head v 40'72, mlcod=40'70)", 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.223805837999997, 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595333, 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527601+0000", 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.522e-06 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536775+0000", 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091746569999999993 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536818+0000", 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2435000000000003e-05 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536828+0000", 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0624e-05 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536931+0000", 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010280000000000001 2024-09-16T13:17:59.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536961+0000", 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0060999999999999e-05 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544146+0000", 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071846339999999996 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544195+0000", 2024-09-16T13:17:59.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8600000000000002e-05 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13719 3.19 3:99fbe691:::rbd_data.112df1cc59bd.000000000000000c:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.470636267000003, 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011713708999999999, 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13719 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280771+0000", 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9240000000000001e-06 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280782+0000", 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1642e-05 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280806+0000", 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4119000000000001e-05 2024-09-16T13:17:59.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280815+0000", 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7150000000000004e-06 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012396900000000001 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281142+0000", 2024-09-16T13:17:59.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000203298 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281540+0000", 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039763500000000002 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.286598+0000", 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050579090000000002 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:59.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292380+0000", 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057823390000000001 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292433+0000", 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2612999999999999e-05 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292482+0000", 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9546000000000003e-05 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.771 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13720 3.19 3:99acede8:::rbd_data.112df1cc59bd.000000000000000d:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.470573074999997, 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011830879000000001, 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13720 2024-09-16T13:17:59.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280836+0000", 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.703e-06 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280844+0000", 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280849+0000", 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.254e-06 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282739+0000", 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001889672 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282875+0000", 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013641 2024-09-16T13:17:59.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.283217+0000", 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00034147099999999998 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.288528+0000", 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053110140000000002 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292558+0000", 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040304429999999999 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292620+0000", 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1588999999999996e-05 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292663+0000", 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2922e-05 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.777 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13726 3.19 3:9e45447e:::rbd_data.112df1cc59bd.0000000000000034:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.470472362000002, 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012171434, 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13726 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280934+0000", 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9299999999999996e-07 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280950+0000", 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1141000000000001e-05 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280955+0000", 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5560000000000001e-06 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284230+0000", 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032752789999999999 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284286+0000", 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6456999999999998e-05 2024-09-16T13:17:59.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284430+0000", 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014382899999999999 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.291179+0000", 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067490099999999997 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293014+0000", 2024-09-16T13:17:59.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018347719999999999 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293069+0000", 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5325999999999998e-05 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293104+0000", 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4892999999999999e-05 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14663 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2228224~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115946010000002, 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098295339999999991, 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14663 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635460+0000", 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1500000000000004e-07 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635463+0000", 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.478e-06 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635472+0000", 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4230000000000005e-06 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635498+0000", 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6543e-05 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645115+0000", 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096171710000000007 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645193+0000", 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7245999999999999e-05 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645289+0000", 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5958000000000005e-05 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14664 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115899519999999, 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010033297, 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.788 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14664 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635506+0000", 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5099999999999998e-07 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635511+0000", 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7820000000000001e-06 2024-09-16T13:17:59.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635520+0000", 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5969999999999992e-06 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635529+0000", 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3249999999999997e-06 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645337+0000", 2024-09-16T13:17:59.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098082589999999997 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645427+0000", 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9922999999999999e-05 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645539+0000", 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011156 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.792 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14665 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115870291999997, 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010434005999999999, 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14665 2024-09-16T13:17:59.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635538+0000", 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8329999999999998e-06 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635546+0000", 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0760000000000003e-06 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635551+0000", 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6160000000000004e-06 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645677+0000", 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010126546 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645821+0000", 2024-09-16T13:17:59.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014406799999999999 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645969+0000", 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014730900000000001 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14678 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 0~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115610848000003, 2024-09-16T13:17:59.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012023611999999999, 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14678 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635797+0000", 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7039999999999999e-06 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635810+0000", 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3343000000000001e-05 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635845+0000", 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4751e-05 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635863+0000", 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8210999999999999e-05 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646005+0000", 2024-09-16T13:17:59.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010141766999999999 2024-09-16T13:17:59.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646070+0000", 2024-09-16T13:17:59.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5324e-05 2024-09-16T13:17:59.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647818+0000", 2024-09-16T13:17:59.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017475119999999999 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14679 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1835008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115523142000001, 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012069985, 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.803 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14679 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:17:59.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635883+0000", 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3239999999999999e-06 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635889+0000", 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635905+0000", 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6022999999999999e-05 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635911+0000", 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3740000000000003e-06 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647849+0000", 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011937906999999999 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647895+0000", 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5507999999999999e-05 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647952+0000", 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7132000000000001e-05 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.807 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14680 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2359296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.808 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:59.808 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115486859999997, 2024-09-16T13:17:59.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012172090999999999, 2024-09-16T13:17:59.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.808 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.808 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.808 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.808 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.808 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14680 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635919+0000", 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8299999999999995e-07 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635922+0000", 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.027e-06 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635931+0000", 2024-09-16T13:17:59.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6230000000000003e-06 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635936+0000", 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0069999999999997e-06 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647981+0000", 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012045273 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648033+0000", 2024-09-16T13:17:59.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:17:59.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648090+0000", 2024-09-16T13:17:59.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7154000000000002e-05 2024-09-16T13:17:59.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.812 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14681 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4718592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115463863000002, 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012312528, 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14681 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.813 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635942+0000", 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7599999999999996e-07 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635959+0000", 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6954999999999999e-05 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635968+0000", 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8929999999999994e-06 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635973+0000", 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0170000000000004e-06 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648120+0000", 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012146847000000001 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648170+0000", 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0368000000000003e-05 2024-09-16T13:17:59.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648254+0000", 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3671999999999994e-05 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14694 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115426788999997, 2024-09-16T13:17:59.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012792843999999999, 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14694 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635979+0000", 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0699999999999996e-07 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635982+0000", 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8200000000000001e-06 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635990+0000", 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0239999999999997e-06 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635995+0000", 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648296+0000", 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0123009 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648397+0000", 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101042 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648771+0000", 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037435099999999998 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.821 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14695 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1966080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115404771000001, 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012992136, 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.822 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14695 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636001+0000", 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0999999999999998e-07 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636004+0000", 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5959999999999998e-06 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636012+0000", 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636016+0000", 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9269999999999996e-06 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648804+0000", 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012787543 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648873+0000", 2024-09-16T13:17:59.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9263000000000004e-05 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648992+0000", 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119089 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14696 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2490368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115383080000001, 2024-09-16T13:17:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013132612, 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14696 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636023+0000", 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7300000000000005e-07 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636025+0000", 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.407e-06 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636033+0000", 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0129999999999993e-06 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636042+0000", 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1540000000000008e-06 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649027+0000", 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012985096 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649089+0000", 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1641000000000002e-05 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649155+0000", 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5528e-05 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14697 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.115357179, 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013321177999999999, 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14697 2024-09-16T13:17:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636049+0000", 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636051+0000", 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.57e-06 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636059+0000", 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9440000000000005e-06 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636067+0000", 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0460000000000006e-06 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649182+0000", 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01311522 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649245+0000", 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2930999999999998e-05 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649369+0000", 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123681 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:29334 3.0 3:063d9f84:::rbd_data.112dab4f0555.0000000000000034:1b [read 1474560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:56.285631+0000", 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.4657739969999999, 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096785840000000005, 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 29334 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285631+0000", 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285631+0000", 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285631+0000", 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8099999999999998e-07 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285634+0000", 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3750000000000001e-06 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285643+0000", 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0210000000000005e-06 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285647+0000", 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3579999999999996e-06 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295177+0000", 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0095294019999999993 2024-09-16T13:17:59.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295247+0000", 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0166999999999994e-05 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295310+0000", 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2879999999999994e-05 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:29335 3.0 3:063d9f84:::rbd_data.112dab4f0555.0000000000000034:1b [read 1671168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:56.285651+0000", 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.4657537019999998, 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0097858489999999992, 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 29335 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285651+0000", 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285651+0000", 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285652+0000", 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285654+0000", 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3329999999999999e-06 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285660+0000", 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2480000000000001e-06 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285664+0000", 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3280000000000002e-06 2024-09-16T13:17:59.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295336+0000", 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096724859999999992 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295390+0000", 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3819000000000001e-05 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295437+0000", 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7216999999999999e-05 2024-09-16T13:17:59.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:29336 3.0 3:063d9f84:::rbd_data.112dab4f0555.0000000000000034:1b [read 1867776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:56.285667+0000", 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.46573756, 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0099780400000000005, 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 29336 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285667+0000", 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285667+0000", 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285668+0000", 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2500000000000001e-07 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285688+0000", 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9958000000000001e-05 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285699+0000", 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1233000000000001e-05 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285704+0000", 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7770000000000002e-06 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295465+0000", 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0097615080000000003 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295536+0000", 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0931000000000002e-05 2024-09-16T13:17:59.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295645+0000", 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109208 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:29337 3.0 3:063d9f84:::rbd_data.112dab4f0555.0000000000000034:1b [read 2064384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:56.285709+0000", 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.4656960290000001, 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010216506, 2024-09-16T13:17:59.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 29337 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285709+0000", 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285709+0000", 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285710+0000", 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285712+0000", 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.39e-06 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285721+0000", 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4630000000000007e-06 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285726+0000", 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295757+0000", 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010031093 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295852+0000", 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5538000000000006e-05 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295925+0000", 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3323000000000003e-05 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:17:59.853 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:17:59.853 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:59.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:17:59.855 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:17:59.848+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:17:59.955 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:17:59.951+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:17:59.958 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:17:59.958 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:17:59.958 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:17:59.958 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:17:59.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:00.056 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:00.048+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:00.063 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:00.063 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:00.064 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:00.064 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:00.064 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:00.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:00.096 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:18:00.157 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:00.148+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:00.172 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:00.172 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:00.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.160300134000003, 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:18:00.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:18:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:18:00.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.159514635000001, 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:18:00.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.181 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:18:00.182 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:00.182 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.158347712999998, 2024-09-16T13:18:00.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:18:00.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.182 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:00.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:00.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:18:00.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:18:00.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:18:00.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:18:00.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.186 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.186 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:18:00.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:18:00.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:18:00.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:18:00.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:18:00.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:18:00.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:18:00.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:18:00.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:18:00.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:18:00.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:18:00.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:18:00.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.191 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.191 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.191 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:18:00.191 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:00.192 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.155473082, 2024-09-16T13:18:00.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:18:00.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.192 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:00.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:18:00.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:18:00.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:18:00.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:18:00.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:18:00.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:18:00.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:18:00.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:18:00.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.198 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.199 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:18:00.199 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:00.199 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.152459342, 2024-09-16T13:18:00.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:18:00.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.200 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.200 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:00.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:18:00.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.127255138999999, 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:18:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:18:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:18:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:18:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 55.361106249999999, 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:18:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:18:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:18:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5115 3.4 e34/23 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head v 34'14, mlcod=31'12)", 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.376058143999998, 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029400680000000001, 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791635+0000", 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4200000000000005e-07 2024-09-16T13:18:00.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791643+0000", 2024-09-16T13:18:00.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.157e-06 2024-09-16T13:18:00.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791660+0000", 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6246e-05 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791666+0000", 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791743+0000", 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7000000000000001e-05 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791756+0000", 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2729e-05 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794523+0000", 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027675949999999999 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794575+0000", 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1082000000000003e-05 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.109003946999998, 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:18:00.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:18:00.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:18:00.222 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.108394507, 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:00.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:18:00.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 47.611110693000001, 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.227 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:18:00.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:18:00.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:18:00.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:18:00.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:18:00.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:18:00.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:18:00.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 47.610009877000003, 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.232 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:18:00.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:18:00.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 47.609870237000003, 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.240 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:18:00.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:18:00.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.240 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:18:00.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:18:00.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:18:00.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:18:00.242 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.242 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.242 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.242 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 47.609369706999999, 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:18:00.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:18:00.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.614200320999998, 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:18:00.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:18:00.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.357786480000001, 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:18:00.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:18:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:18:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:18:00.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:18:00.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:00.256 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 31.745223373000002, 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:18:00.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.260 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:22854 3.9 e57/23 3:924a2914:::rbd_data.112db5030c67.0000000000000008:head v 57'198, mlcod=57'196)", 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.626702809000001, 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029464980000000001, 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540995+0000", 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541018+0000", 2024-09-16T13:18:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2382999999999999e-05 2024-09-16T13:18:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541079+0000", 2024-09-16T13:18:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1773999999999995e-05 2024-09-16T13:18:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541094+0000", 2024-09-16T13:18:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4414e-05 2024-09-16T13:18:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.542159+0000", 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010653749999999999 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.542208+0000", 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8517000000000003e-05 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.543797+0000", 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015893210000000001 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.543936+0000", 2024-09-16T13:18:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000139367 2024-09-16T13:18:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:18:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.9071172030000003, 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:00.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:18:00.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.5429898699999995, 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:18:00.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:18:00.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.275 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:00.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:00.277 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:00.268+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:00.347 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone26 from 0xbe11db5 (overlap 0x2926ac3) to 0x18b0cdb 2024-09-16T13:18:00.378 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:00.368+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:00.436 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:00.436 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:00.436 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:00.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:00.495 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:00.488+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:00.548 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:00.548 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:00.549 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:00.549 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:00.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:00.579 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:00.571+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:00.659 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:00.659 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:00.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5101 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.876450706, 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595248, 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5101 2024-09-16T13:18:00.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777329+0000", 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777333+0000", 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1949999999999996e-06 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777342+0000", 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6950000000000006e-06 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777348+0000", 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5500000000000002e-06 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785241+0000", 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0078929329999999995 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785391+0000", 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000150113 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.785969+0000", 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057822899999999998 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789745+0000", 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037756529999999999 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793823+0000", 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040786299999999998 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793880+0000", 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7157999999999997e-05 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793924+0000", 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3407e-05 2024-09-16T13:18:00.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.666 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.666 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5102 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.666 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:18:00.666 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.876403414000002, 2024-09-16T13:18:00.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016663710000000002, 2024-09-16T13:18:00.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5102 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777376+0000", 2024-09-16T13:18:00.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777377+0000", 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4600000000000003e-07 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777381+0000", 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5290000000000002e-06 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777393+0000", 2024-09-16T13:18:00.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2001000000000001e-05 2024-09-16T13:18:00.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777401+0000", 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3869999999999997e-06 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786753+0000", 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0093523649999999996 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.786860+0000", 2024-09-16T13:18:00.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106921 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.787286+0000", 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042651800000000003 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793815+0000", 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0065290579999999999 2024-09-16T13:18:00.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793932+0000", 2024-09-16T13:18:00.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116688 2024-09-16T13:18:00.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794002+0000", 2024-09-16T13:18:00.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0313000000000003e-05 2024-09-16T13:18:00.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794040+0000", 2024-09-16T13:18:00.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7083999999999999e-05 2024-09-16T13:18:00.674 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.674 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.674 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.674 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5103 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.675 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:18:00.675 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.876342528000002, 2024-09-16T13:18:00.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017505544000000001, 2024-09-16T13:18:00.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.675 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.676 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.676 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5103 2024-09-16T13:18:00.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.676 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:18:00.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777437+0000", 2024-09-16T13:18:00.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777438+0000", 2024-09-16T13:18:00.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.78e-07 2024-09-16T13:18:00.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777442+0000", 2024-09-16T13:18:00.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5950000000000001e-06 2024-09-16T13:18:00.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777453+0000", 2024-09-16T13:18:00.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0372e-05 2024-09-16T13:18:00.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777458+0000", 2024-09-16T13:18:00.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6559999999999997e-06 2024-09-16T13:18:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788034+0000", 2024-09-16T13:18:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010575753 2024-09-16T13:18:00.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788111+0000", 2024-09-16T13:18:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7038000000000003e-05 2024-09-16T13:18:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788463+0000", 2024-09-16T13:18:00.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000352028 2024-09-16T13:18:00.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794263+0000", 2024-09-16T13:18:00.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005800342 2024-09-16T13:18:00.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794837+0000", 2024-09-16T13:18:00.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00057334300000000003 2024-09-16T13:18:00.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794897+0000", 2024-09-16T13:18:00.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0690000000000003e-05 2024-09-16T13:18:00.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794942+0000", 2024-09-16T13:18:00.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4848999999999997e-05 2024-09-16T13:18:00.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.689 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5104 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.689 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:18:00.689 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.876298120000001, 2024-09-16T13:18:00.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017694142999999999, 2024-09-16T13:18:00.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.690 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.690 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.690 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.690 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5104 2024-09-16T13:18:00.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.691 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:18:00.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777481+0000", 2024-09-16T13:18:00.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777482+0000", 2024-09-16T13:18:00.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0499999999999999e-06 2024-09-16T13:18:00.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777487+0000", 2024-09-16T13:18:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6639999999999997e-06 2024-09-16T13:18:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777496+0000", 2024-09-16T13:18:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2890000000000003e-06 2024-09-16T13:18:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777501+0000", 2024-09-16T13:18:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2730000000000002e-06 2024-09-16T13:18:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.788951+0000", 2024-09-16T13:18:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011449429000000001 2024-09-16T13:18:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789007+0000", 2024-09-16T13:18:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5643999999999998e-05 2024-09-16T13:18:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789205+0000", 2024-09-16T13:18:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019820800000000001 2024-09-16T13:18:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794268+0000", 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050636850000000001 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795094+0000", 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00082569700000000002 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795139+0000", 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5055000000000001e-05 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795175+0000", 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6149000000000002e-05 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5105 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.876235778999998, 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017729482000000001, 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5105 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777543+0000", 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777544+0000", 2024-09-16T13:18:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1100000000000004e-07 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777549+0000", 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777561+0000", 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1524e-05 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777566+0000", 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.558e-06 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789781+0000", 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012214981 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.789870+0000", 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9171000000000004e-05 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790313+0000", 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044255299999999999 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794273+0000", 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039596129999999998 2024-09-16T13:18:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795213+0000", 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00094086099999999998 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795239+0000", 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5324e-05 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.795273+0000", 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4198e-05 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5106 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.876190635, 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018621867, 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5106 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777589+0000", 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777590+0000", 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777594+0000", 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6750000000000001e-06 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777604+0000", 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0076e-05 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777610+0000", 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7529999999999998e-06 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790893+0000", 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013282845 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.790999+0000", 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000106289 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791493+0000", 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00049357800000000005 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794662+0000", 2024-09-16T13:18:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031691409999999999 2024-09-16T13:18:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796146+0000", 2024-09-16T13:18:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00148375 2024-09-16T13:18:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796176+0000", 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0349e-05 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796211+0000", 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4530000000000003e-05 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5107 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:18:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.876159373999997, 2024-09-16T13:18:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018745067000000001, 2024-09-16T13:18:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5107 2024-09-16T13:18:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:18:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777620+0000", 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777621+0000", 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0900000000000001e-07 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777624+0000", 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7950000000000001e-06 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777633+0000", 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5979999999999997e-06 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777638+0000", 2024-09-16T13:18:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.989e-06 2024-09-16T13:18:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792024+0000", 2024-09-16T13:18:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014385692 2024-09-16T13:18:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792104+0000", 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0567000000000003e-05 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792340+0000", 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023624200000000001 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794667+0000", 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0023269990000000002 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796304+0000", 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001636981 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796331+0000", 2024-09-16T13:18:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6492999999999999e-05 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796365+0000", 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4102e-05 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5108 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.722 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.876105545000001, 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019090617000000001, 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5108 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:18:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777674+0000", 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6000000000000004e-07 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777677+0000", 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7879999999999998e-06 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777686+0000", 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7679999999999997e-06 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.777689+0000", 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.123e-06 2024-09-16T13:18:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792751+0000", 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01506158 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.792855+0000", 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010456 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.793305+0000", 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00045006100000000003 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794671+0000", 2024-09-16T13:18:00.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001366304 2024-09-16T13:18:00.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796393+0000", 2024-09-16T13:18:00.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001721272 2024-09-16T13:18:00.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796733+0000", 2024-09-16T13:18:00.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000340611 2024-09-16T13:18:00.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.796764+0000", 2024-09-16T13:18:00.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.099e-05 2024-09-16T13:18:00.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.729 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5208 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.730 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:18:00.730 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.774346094000002, 2024-09-16T13:18:00.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014882177999999999, 2024-09-16T13:18:00.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.730 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.731 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.731 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.731 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5208 2024-09-16T13:18:00.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.731 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:18:00.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879433+0000", 2024-09-16T13:18:00.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879434+0000", 2024-09-16T13:18:00.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.322e-06 2024-09-16T13:18:00.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879443+0000", 2024-09-16T13:18:00.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3459999999999994e-06 2024-09-16T13:18:00.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879459+0000", 2024-09-16T13:18:00.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6133999999999998e-05 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879467+0000", 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2400000000000007e-06 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888339+0000", 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0088721619999999994 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888461+0000", 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000121341 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.888819+0000", 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035782799999999998 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891271+0000", 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002452872 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894224+0000", 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029522910000000001 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894275+0000", 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1647999999999997e-05 2024-09-16T13:18:00.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894315+0000", 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9993999999999997e-05 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5210 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.773790906000002, 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01466423, 2024-09-16T13:18:00.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5210 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879988+0000", 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879989+0000", 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8800000000000001e-07 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.879995+0000", 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3959999999999997e-06 2024-09-16T13:18:00.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880017+0000", 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2101000000000001e-05 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880024+0000", 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9240000000000001e-06 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890250+0000", 2024-09-16T13:18:00.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010226012 2024-09-16T13:18:00.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890344+0000", 2024-09-16T13:18:00.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.446e-05 2024-09-16T13:18:00.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.890770+0000", 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00042575099999999998 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894344+0000", 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003574163 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894549+0000", 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020477599999999999 2024-09-16T13:18:00.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894606+0000", 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6891000000000001e-05 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894653+0000", 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6867999999999999e-05 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5211 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.773746842999998, 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014800604, 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5211 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880032+0000", 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880033+0000", 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.06e-07 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880039+0000", 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5840000000000002e-06 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880052+0000", 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3519999999999999e-05 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880059+0000", 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3090000000000001e-06 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891316+0000", 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011257118999999999 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891374+0000", 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8020999999999999e-05 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.891577+0000", 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020371699999999999 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894348+0000", 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027708300000000002 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894726+0000", 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037819399999999999 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894784+0000", 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7667999999999997e-05 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894833+0000", 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8936e-05 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5212 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.750 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.773391469000003, 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014645704000000001, 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5212 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880388+0000", 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880389+0000", 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1680000000000001e-06 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880395+0000", 2024-09-16T13:18:00.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880408+0000", 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.308e-05 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880414+0000", 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.105e-06 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892037+0000", 2024-09-16T13:18:00.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011622402 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892116+0000", 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9939999999999997e-05 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.892597+0000", 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048076500000000002 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894492+0000", 2024-09-16T13:18:00.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001895246 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894921+0000", 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000428114 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894985+0000", 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4777999999999995e-05 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895034+0000", 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8133999999999997e-05 2024-09-16T13:18:00.755 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5213 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.773258319, 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014632608, 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5213 2024-09-16T13:18:00.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880521+0000", 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880522+0000", 2024-09-16T13:18:00.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2600000000000001e-07 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880527+0000", 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880538+0000", 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0917999999999999e-05 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880543+0000", 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.361e-06 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893069+0000", 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012525497999999999 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893152+0000", 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2805000000000004e-05 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.893635+0000", 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048315100000000002 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.894852+0000", 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001216806 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895075+0000", 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022300299999999999 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895122+0000", 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7077999999999999e-05 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895154+0000", 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1919999999999999e-05 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5214 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:18:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.773103667999997, 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017785977000000001, 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5214 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880676+0000", 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8100000000000001e-07 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880681+0000", 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880689+0000", 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3610000000000001e-06 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880694+0000", 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5809999999999998e-06 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895186+0000", 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014492430000000001 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895311+0000", 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012518300000000001 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.895760+0000", 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044864900000000002 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898337+0000", 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0025765229999999998 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898354+0000", 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7538999999999999e-05 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898406+0000", 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2054e-05 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898462+0000", 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5404e-05 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5215 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.772838833999998, 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018191319000000001, 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5215 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.767 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880940+0000", 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880941+0000", 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2399999999999997e-07 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880946+0000", 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3710000000000002e-06 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880955+0000", 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1230000000000006e-06 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.880959+0000", 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7879999999999997e-06 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896782+0000", 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015822716000000001 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.896876+0000", 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3667999999999998e-05 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.897289+0000", 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00041357399999999999 2024-09-16T13:18:00.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.898474+0000", 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011847209999999999 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899036+0000", 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00056215699999999998 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899090+0000", 2024-09-16T13:18:00.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3662999999999998e-05 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.899132+0000", 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1814000000000003e-05 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5282 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.704166720000003, 2024-09-16T13:18:00.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014924988, 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5282 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949613+0000", 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.53e-07 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949635+0000", 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2422999999999999e-05 2024-09-16T13:18:00.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949641+0000", 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.896e-06 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949644+0000", 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5270000000000001e-06 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959656+0000", 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010012378000000001 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.959747+0000", 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1273000000000005e-05 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960045+0000", 2024-09-16T13:18:00.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00029733 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963196+0000", 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031512620000000002 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964453+0000", 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001257357 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964502+0000", 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9044e-05 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964538+0000", 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5244999999999999e-05 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5283 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.778 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.704131566000001, 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015065762, 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5283 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949648+0000", 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4799999999999999e-07 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949650+0000", 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.736e-06 2024-09-16T13:18:00.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949653+0000", 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2969999999999999e-06 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949655+0000", 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1890000000000001e-06 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960643+0000", 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010988015 2024-09-16T13:18:00.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.960732+0000", 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9097000000000004e-05 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961195+0000", 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000462292 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963208+0000", 2024-09-16T13:18:00.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.002013641 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964585+0000", 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0013765769999999999 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964669+0000", 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3863000000000007e-05 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964713+0000", 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4706999999999997e-05 2024-09-16T13:18:00.783 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5284 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.704121696999998, 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015152802, 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5284 2024-09-16T13:18:00.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949658+0000", 2024-09-16T13:18:00.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5899999999999998e-07 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949659+0000", 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.601e-06 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949672+0000", 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2364999999999999e-05 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949674+0000", 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:18:00.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961869+0000", 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012194903999999999 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.961971+0000", 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000102154 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.962502+0000", 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00053037500000000003 2024-09-16T13:18:00.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964751+0000", 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0022487800000000001 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964755+0000", 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964789+0000", 2024-09-16T13:18:00.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4054000000000003e-05 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964810+0000", 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0922000000000001e-05 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:5285 3.b 3:d7ac1647:::rbd_object_map.112d64676e7f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.704102779000003, 2024-09-16T13:18:00.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.015553022, 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 5285 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949676+0000", 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949677+0000", 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6899999999999999e-07 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949679+0000", 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0090000000000002e-06 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949682+0000", 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2550000000000002e-06 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.949685+0000", 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.621e-06 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963409+0000", 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013724485999999999 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963497+0000", 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7560999999999997e-05 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.963933+0000", 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00043592700000000002 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.964820+0000", 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00088772299999999998 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965165+0000", 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000344426 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965201+0000", 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5967000000000003e-05 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.965230+0000", 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8778000000000001e-05 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11223 3.8 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 33927~20122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.192930449000002, 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.019333487, 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:00.795 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11223 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460849+0000", 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.460851+0000", 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5730000000000002e-06 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464824+0000", 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039725730000000001 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464861+0000", 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7233000000000001e-05 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464872+0000", 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1209000000000001e-05 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.464987+0000", 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011423 2024-09-16T13:18:00.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465188+0000", 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020091600000000001 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.465929+0000", 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00074102 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.475697+0000", 2024-09-16T13:18:00.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.009768367 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480068+0000", 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043713290000000002 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480143+0000", 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5000999999999996e-05 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:00.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.480182+0000", 2024-09-16T13:18:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9035999999999997e-05 2024-09-16T13:18:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:00.801 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:00.801 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:00.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:00.803 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #25, image image_client.0-clone25 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent26 2024-09-16T13:18:00.803 INFO:teuthology.orchestra.run.smithi098.stdout:4166 read 0x4b2bcef thru 0x4b367af (0xaac1 bytes) 2024-09-16T13:18:00.803 INFO:teuthology.orchestra.run.smithi098.stdout:4167 read 0xbacef90 thru 0xbad7b38 (0x8ba9 bytes) 2024-09-16T13:18:00.803 INFO:teuthology.orchestra.run.smithi098.stdout:4168 read 0x4741e0 thru 0x476183 (0x1fa4 bytes) 2024-09-16T13:18:00.804 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:00.792+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:00.804 INFO:teuthology.orchestra.run.smithi098.stdout:4169 trunc from 0xbe11db5 to 0x14e3dc5 2024-09-16T13:18:00.903 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:00.895+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:00.904 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:00.905 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:00.905 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:00.905 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:00.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:01.004 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:01.000+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:01.011 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:01.011 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:01.011 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:01.011 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:01.011 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:01.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:01.105 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:01.099+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:18:01.133 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:01.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8369 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.071159254999998, 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020058919000000001, 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8369 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056335+0000", 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056338+0000", 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056344+0000", 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8150000000000002e-06 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056348+0000", 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.070951+0000", 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014603055 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071060+0000", 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109462 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071508+0000", 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044824200000000001 2024-09-16T13:18:01.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074304+0000", 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027951719999999998 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076331+0000", 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020276529999999999 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076369+0000", 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7557e-05 2024-09-16T13:18:01.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076393+0000", 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4445999999999999e-05 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8370 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.071141937, 2024-09-16T13:18:01.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021240851000000002, 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8370 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056355+0000", 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.667e-06 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056360+0000", 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6620000000000002e-06 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056366+0000", 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072672+0000", 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016306157000000002 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072746+0000", 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3648999999999999e-05 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.073364+0000", 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061885499999999997 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076445+0000", 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003080396 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077482+0000", 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001036709 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077535+0000", 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3211000000000003e-05 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077592+0000", 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7664000000000002e-05 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8371 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.071113142000002, 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021386721000000001, 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8371 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.145 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056381+0000", 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-16T13:18:01.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056385+0000", 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056391+0000", 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6959999999999999e-06 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056396+0000", 2024-09-16T13:18:01.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2549999999999999e-06 2024-09-16T13:18:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074355+0000", 2024-09-16T13:18:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017959098999999999 2024-09-16T13:18:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074503+0000", 2024-09-16T13:18:01.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014797200000000001 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074955+0000", 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045213300000000002 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077607+0000", 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026519500000000001 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077660+0000", 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077725+0000", 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4991000000000002e-05 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077767+0000", 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:18:01.150 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8372 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.071092121, 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021557531000000001, 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056402+0000", 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1399999999999997e-07 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056405+0000", 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6520000000000002e-06 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056411+0000", 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2009999999999996e-06 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056415+0000", 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.211e-06 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075482+0000", 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019067163000000002 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075546+0000", 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3516000000000006e-05 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075844+0000", 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029810999999999998 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077784+0000", 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019399350000000001 2024-09-16T13:18:01.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077823+0000", 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9524000000000003e-05 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077917+0000", 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4090000000000002e-05 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077959+0000", 2024-09-16T13:18:01.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1615000000000001e-05 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8373 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.071071777999997, 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022897421000000001, 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.157 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3200000000000005e-07 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056427+0000", 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-06 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056433+0000", 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056437+0000", 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076470+0000", 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020032239 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076524+0000", 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4475000000000003e-05 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076996+0000", 2024-09-16T13:18:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000471905 2024-09-16T13:18:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.078749+0000", 2024-09-16T13:18:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001752539 2024-09-16T13:18:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079217+0000", 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046819400000000001 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079276+0000", 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9407000000000002e-05 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079319+0000", 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.162 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8398 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.041707985999999, 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017439282, 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8398 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.163 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:18:01.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085788+0000", 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4719999999999998e-06 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085793+0000", 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6179999999999997e-06 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085796+0000", 2024-09-16T13:18:01.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6680000000000001e-06 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099514+0000", 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013717895000000001 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099873+0000", 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035970799999999998 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100269+0000", 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000395382 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103068+0000", 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027992680000000002 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103114+0000", 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6189000000000001e-05 2024-09-16T13:18:01.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103178+0000", 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3845999999999998e-05 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103225+0000", 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6888999999999998e-05 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.168 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8399 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.041666509000002, 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017551573000000001, 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8399 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.169 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085828+0000", 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-16T13:18:01.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085830+0000", 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8609999999999998e-06 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085835+0000", 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.437e-06 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085837+0000", 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.571e-06 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100950+0000", 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015112537000000001 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101047+0000", 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6945000000000001e-05 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101468+0000", 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000420794 2024-09-16T13:18:01.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103315+0000", 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018474769999999999 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103321+0000", 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103356+0000", 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4764000000000001e-05 2024-09-16T13:18:01.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103379+0000", 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3102999999999999e-05 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8400 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.041653607999997, 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017960752, 2024-09-16T13:18:01.174 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8400 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.53e-07 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085842+0000", 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2620000000000001e-06 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085847+0000", 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2119999999999997e-06 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085849+0000", 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.204e-06 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101995+0000", 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016146384999999999 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102100+0000", 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010482199999999999 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102466+0000", 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000365731 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103389+0000", 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092367199999999995 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103702+0000", 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031220400000000001 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103754+0000", 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2318999999999998e-05 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103801+0000", 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6687999999999998e-05 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8539 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.883522053, 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018486425000000001, 2024-09-16T13:18:01.180 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8539 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0500000000000003e-07 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243975+0000", 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800000000000002e-06 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243982+0000", 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243986+0000", 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4100000000000001e-06 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258523+0000", 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014536604999999999 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258678+0000", 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155341 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.259218+0000", 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053930600000000003 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262325+0000", 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031076799999999998 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262367+0000", 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1480000000000003e-05 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262417+0000", 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0555e-05 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262458+0000", 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0556999999999998e-05 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8540 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.883502632000003, 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018906683000000001, 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8540 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243992+0000", 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.75e-07 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243995+0000", 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1219999999999999e-06 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244001+0000", 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5010000000000001e-06 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244006+0000", 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999996e-06 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260137+0000", 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016131 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260200+0000", 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3782999999999996e-05 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260598+0000", 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039746600000000002 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262331+0000", 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017327849999999999 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262786+0000", 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045546300000000003 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262851+0000", 2024-09-16T13:18:01.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-05 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262898+0000", 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7098000000000002e-05 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8541 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.883483409, 2024-09-16T13:18:01.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019366889000000002, 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8541 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244011+0000", 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244014+0000", 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.79e-06 2024-09-16T13:18:01.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244020+0000", 2024-09-16T13:18:01.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6370000000000002e-06 2024-09-16T13:18:01.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244025+0000", 2024-09-16T13:18:01.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3359999999999997e-06 2024-09-16T13:18:01.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261122+0000", 2024-09-16T13:18:01.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017097161 2024-09-16T13:18:01.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261228+0000", 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105989 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261475+0000", 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024757000000000002 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263022+0000", 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015466760000000001 2024-09-16T13:18:01.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263300+0000", 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027828100000000001 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263344+0000", 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4008999999999998e-05 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263377+0000", 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2832e-05 2024-09-16T13:18:01.198 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11776 3.1a 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 43.610080402000001, 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027432719000000001, 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.199 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11776 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517415+0000", 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5239999999999999e-06 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526328+0000", 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089138059999999998 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526362+0000", 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3775000000000001e-05 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526379+0000", 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6835000000000001e-05 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526467+0000", 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7703999999999999e-05 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526653+0000", 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186379 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.527624+0000", 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097080400000000002 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543313+0000", 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015689456000000001 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544696+0000", 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013822979999999999 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544784+0000", 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8419999999999997e-05 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544846+0000", 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1717999999999994e-05 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:11778 3.1 e40/23 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head v 40'57, mlcod=40'55)", 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 43.603596375000002, 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01745787, 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523898+0000", 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-06 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533067+0000", 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091691120000000001 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533101+0000", 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3733999999999999e-05 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533115+0000", 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3410999999999999e-05 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533194+0000", 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9827000000000007e-05 2024-09-16T13:18:01.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533223+0000", 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8937000000000001e-05 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541297+0000", 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080739850000000005 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541355+0000", 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7708999999999999e-05 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11777 3.1a 3:59cb7a3f:::rbd_object_map.112d21cb8818:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 43.601091464, 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018649718999999999, 2024-09-16T13:18:01.209 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11777 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526404+0000", 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.654e-06 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526409+0000", 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.882e-06 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526424+0000", 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5328999999999998e-05 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526428+0000", 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6219999999999998e-06 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542209+0000", 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015781765999999999 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542334+0000", 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125148 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542804+0000", 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046944899999999998 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543732+0000", 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092824699999999999 2024-09-16T13:18:01.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544929+0000", 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001196522 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545004+0000", 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4962999999999994e-05 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545052+0000", 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8136999999999997e-05 2024-09-16T13:18:01.214 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11779 3.9 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 43.580569881999999, 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020806181999999999, 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.215 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11779 2024-09-16T13:18:01.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.216 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:18:01.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546926+0000", 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9769999999999999e-06 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552330+0000", 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054043980000000004 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552353+0000", 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2875999999999999e-05 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552362+0000", 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7299999999999994e-06 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552447+0000", 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5620999999999999e-05 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552710+0000", 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026273300000000002 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.553446+0000", 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073563299999999997 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.559002+0000", 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055564580000000002 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567615+0000", 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086126680000000004 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567689+0000", 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3839999999999998e-05 2024-09-16T13:18:01.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567730+0000", 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1248000000000002e-05 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13736 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.829498704999999, 2024-09-16T13:18:01.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018229967999999999, 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13736 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2e-07 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297998+0000", 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.824e-06 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298004+0000", 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8730000000000004e-06 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298008+0000", 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312651+0000", 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014643351000000001 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312740+0000", 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8466000000000003e-05 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313081+0000", 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034067899999999998 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315982+0000", 2024-09-16T13:18:01.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002900943 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316051+0000", 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9029999999999995e-05 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316101+0000", 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0680000000000003e-05 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316225+0000", 2024-09-16T13:18:01.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012359099999999999 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13737 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.829481585000003, 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018381208999999999, 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13737 2024-09-16T13:18:01.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:18:01.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298015+0000", 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6869999999999999e-06 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298021+0000", 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7799999999999997e-06 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298027+0000", 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.071e-06 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313829+0000", 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015801696 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314019+0000", 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018995700000000001 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:01.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314379+0000", 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036036500000000002 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315988+0000", 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016084949999999999 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316290+0000", 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000302732 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316324+0000", 2024-09-16T13:18:01.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3937999999999999e-05 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316393+0000", 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8966000000000003e-05 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13738 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.829462603000003, 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021400311000000002, 2024-09-16T13:18:01.232 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13738 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298032+0000", 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7400000000000003e-07 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298034+0000", 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5770000000000001e-06 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298040+0000", 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7010000000000001e-06 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298044+0000", 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2069999999999997e-06 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315033+0000", 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016988970999999999 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315106+0000", 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2723000000000002e-05 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315428+0000", 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032191099999999997 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319249+0000", 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003821166 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319313+0000", 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319347+0000", 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4591999999999997e-05 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319431+0000", 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3893000000000005e-05 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.237 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13739 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.829445090999997, 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021506517999999999, 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13739 2024-09-16T13:18:01.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298049+0000", 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298052+0000", 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.819e-06 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298058+0000", 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6960000000000002e-06 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298061+0000", 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7560000000000001e-06 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316465+0000", 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018403744 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316747+0000", 2024-09-16T13:18:01.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028175200000000002 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.317694+0000", 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000947115 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319304+0000", 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016102029999999999 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319446+0000", 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142008 2024-09-16T13:18:01.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319490+0000", 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3690999999999999e-05 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319555+0000", 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5191000000000007e-05 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13740 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:01.243 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.829391608000002, 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023881870999999999, 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13740 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:18:01.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298103+0000", 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298106+0000", 2024-09-16T13:18:01.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1559999999999999e-06 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298120+0000", 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4474999999999999e-05 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298124+0000", 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1699999999999999e-06 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319635+0000", 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021510129999999999 2024-09-16T13:18:01.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319762+0000", 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012746400000000001 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.320225+0000", 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046264800000000002 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321629+0000", 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00140383 2024-09-16T13:18:01.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321682+0000", 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3720000000000001e-05 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321757+0000", 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321984+0000", 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022684199999999999 2024-09-16T13:18:01.248 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.249 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.249 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.249 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:01.249 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:01.249 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:01.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:01.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:01.252 INFO:teuthology.orchestra.run.smithi098.stdout:4171 trunc from 0x14e3dc5 to 0xe94790f 2024-09-16T13:18:01.252 INFO:teuthology.orchestra.run.smithi098.stdout:4172 read 0x1334fc9 thru 0x1336b87 (0x1bbf bytes) 2024-09-16T13:18:01.252 INFO:teuthology.orchestra.run.smithi098.stdout:4174 writesame 0x3fed5ec thru 0x3ff8256 data_size 0x1(0xac6b bytes) 2024-09-16T13:18:01.252 INFO:teuthology.orchestra.run.smithi098.stdout:4175 read 0x82d79c7 thru 0x82e6db6 (0xf3f0 bytes) 2024-09-16T13:18:01.252 INFO:teuthology.orchestra.run.smithi098.stdout:4176 write 0x23b1c6d thru 0x23c0068 (0xe3fc bytes) 2024-09-16T13:18:01.252 INFO:teuthology.orchestra.run.smithi098.stdout:4177 writesame 0xdc0f0cd thru 0xdc17083 data_size 0xd(0x7fb7 bytes) 2024-09-16T13:18:01.252 INFO:teuthology.orchestra.run.smithi098.stdout:4178 write 0x8b7538f thru 0x8b80d23 (0xb995 bytes) 2024-09-16T13:18:01.252 INFO:teuthology.orchestra.run.smithi098.stdout:4180 write 0x8000324 thru 0x80074a0 (0x717d bytes) 2024-09-16T13:18:01.252 INFO:teuthology.orchestra.run.smithi098.stdout:4181 write 0x9391d04 thru 0x939d115 (0xb412 bytes) 2024-09-16T13:18:01.253 INFO:teuthology.orchestra.run.smithi098.stdout:4182 writesame 0xd69d59d thru 0xd6aaab6 data_size 0x1(0xd51a bytes) 2024-09-16T13:18:01.253 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:01.240+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:01.284 INFO:teuthology.orchestra.run.smithi098.stdout:4185 punch from 0x4b98e02 to 0x4b9ebd5, (0x5dd3 bytes) 2024-09-16T13:18:01.284 INFO:teuthology.orchestra.run.smithi098.stdout:4186 read 0xad51923 thru 0xad5a45a (0x8b38 bytes) 2024-09-16T13:18:01.284 INFO:teuthology.orchestra.run.smithi098.stdout:4187 write 0xaa3868 thru 0xab1859 (0xdff2 bytes) 2024-09-16T13:18:01.296 INFO:teuthology.orchestra.run.smithi098.stdout:4188 read 0x80d06c9 thru 0x80db1a8 (0xaae0 bytes) 2024-09-16T13:18:01.296 INFO:teuthology.orchestra.run.smithi098.stdout:4189 punch from 0x8371267 to 0x83772e1, (0x607a bytes) 2024-09-16T13:18:01.296 INFO:teuthology.orchestra.run.smithi098.stdout:4190 punch from 0x2983fd to 0x2a646e, (0xe071 bytes) 2024-09-16T13:18:01.297 INFO:teuthology.orchestra.run.smithi098.stdout:4191 read 0xac5059 thru 0xad41ee (0xf196 bytes) 2024-09-16T13:18:01.297 INFO:teuthology.orchestra.run.smithi098.stdout:4192 read 0xb5e4413 thru 0xb5f3d4d (0xf93b bytes) 2024-09-16T13:18:01.297 INFO:teuthology.orchestra.run.smithi098.stdout:4194 read 0x591e410 thru 0x5921a02 (0x35f3 bytes) 2024-09-16T13:18:01.307 INFO:teuthology.orchestra.run.smithi098.stdout:4195 trunc from 0xe94790f to 0x8d0baa8 2024-09-16T13:18:01.336 INFO:teuthology.orchestra.run.smithi098.stdout:4197 trunc from 0x8d0baa8 to 0xa7e802f 2024-09-16T13:18:01.348 INFO:teuthology.orchestra.run.smithi098.stdout:4199 read 0x7d1bfb1 thru 0x7d2bee5 (0xff35 bytes) 2024-09-16T13:18:01.348 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:01.348 INFO:teuthology.orchestra.run.smithi098.stdout:4201 punch from 0x5b36a95 to 0x5b3fca8, (0x9213 bytes) 2024-09-16T13:18:01.348 INFO:teuthology.orchestra.run.smithi098.stdout:4202 punch from 0x7e690fd to 0x7e7510a, (0xc00d bytes) 2024-09-16T13:18:01.349 INFO:teuthology.orchestra.run.smithi098.stdout:4203 read 0x6676154 thru 0x6685873 (0xf720 bytes) 2024-09-16T13:18:01.349 INFO:teuthology.orchestra.run.smithi098.stdout:4204 read 0x8e0fbda thru 0x8e1e5df (0xea06 bytes) 2024-09-16T13:18:01.349 INFO:teuthology.orchestra.run.smithi098.stdout:4205 writesame 0x552c339 thru 0x553737b data_size 0x59(0xb043 bytes) 2024-09-16T13:18:01.351 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:01.343+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:01.358 INFO:teuthology.orchestra.run.smithi098.stdout:4206 write 0x6f3921f thru 0x6f3ed7e (0x5b60 bytes) 2024-09-16T13:18:01.362 INFO:teuthology.orchestra.run.smithi098.stdout:4207 writesame 0x4d81583 thru 0x4d90a6b data_size 0x1(0xf4e9 bytes) 2024-09-16T13:18:01.392 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:01.402 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:01.402 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:01.402 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:01.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:01.413 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:01.416 INFO:teuthology.orchestra.run.smithi098.stdout:4209 write 0x20cdd22 thru 0x20d8385 (0xa664 bytes) 2024-09-16T13:18:01.423 INFO:teuthology.orchestra.run.smithi098.stdout:4210 read 0x5316f4b thru 0x531f28d (0x8343 bytes) 2024-09-16T13:18:01.423 INFO:teuthology.orchestra.run.smithi098.stdout:4211 write 0x106b2df thru 0x106fd9f (0x4ac1 bytes) 2024-09-16T13:18:01.429 INFO:teuthology.orchestra.run.smithi098.stdout:4212 punch from 0x6eea93a to 0x6ef0188, (0x584e bytes) 2024-09-16T13:18:01.429 INFO:teuthology.orchestra.run.smithi098.stdout:4213 punch from 0x3309955 to 0x330e6ee, (0x4d99 bytes) 2024-09-16T13:18:01.432 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:01.433 INFO:teuthology.orchestra.run.smithi098.stdout:4214 trunc from 0xa7e802f to 0xcd7aaab 2024-09-16T13:18:01.441 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:01.441 INFO:teuthology.orchestra.run.smithi098.stdout:4216 write 0xc4cb7c0 thru 0xc4d6e7e (0xb6bf bytes) 2024-09-16T13:18:01.451 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:01.452 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:01.444+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:01.467 INFO:teuthology.orchestra.run.smithi098.stdout:4219 trunc from 0xcd7aaab to 0x9da86d 2024-09-16T13:18:01.473 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:01.503 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:01.507 INFO:teuthology.orchestra.run.smithi098.stdout:4220 read 0x2506bd thru 0x2559bd (0x5301 bytes) 2024-09-16T13:18:01.508 INFO:teuthology.orchestra.run.smithi098.stdout:4221 read 0x57e2b7 thru 0x57e33c (0x86 bytes) 2024-09-16T13:18:01.519 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:01.519 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:01.525 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:01.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:01.532 INFO:teuthology.orchestra.run.smithi098.stdout:4222 write 0x8978cff thru 0x897ffce (0x72d0 bytes) 2024-09-16T13:18:01.538 INFO:teuthology.orchestra.run.smithi098.stdout:4223 read 0x31f8de2 thru 0x3203475 (0xa694 bytes) 2024-09-16T13:18:01.538 INFO:teuthology.orchestra.run.smithi098.stdout:4227 writesame 0x5e50134 thru 0x5e502c6 data_size 0x1(0x193 bytes) 2024-09-16T13:18:01.544 INFO:teuthology.orchestra.run.smithi098.stdout:4230 read 0x8134bc thru 0x81b555 (0x809a bytes) 2024-09-16T13:18:01.546 INFO:teuthology.orchestra.run.smithi098.stdout:4232 read 0x263fc6 thru 0x26be0c (0x7e47 bytes) 2024-09-16T13:18:01.553 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:01.553 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:01.544+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:01.554 INFO:teuthology.orchestra.run.smithi098.stdout:4233 trunc from 0x897ffcf to 0x526248f 2024-09-16T13:18:01.555 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:01.555 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:01.555 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:01.555 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:01.556 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:01.556 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:01.556 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:01.556 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:18:01.573 INFO:teuthology.orchestra.run.smithi098.stdout:4234 read 0x437291d thru 0x4374c0b (0x22ef bytes) 2024-09-16T13:18:01.584 INFO:teuthology.orchestra.run.smithi098.stdout:4235 trunc from 0x526248f to 0xaed89eb 2024-09-16T13:18:01.597 INFO:teuthology.orchestra.run.smithi098.stdout:4237 write 0x8678d6c thru 0x8683518 (0xa7ad bytes) 2024-09-16T13:18:01.606 INFO:teuthology.orchestra.run.smithi098.stdout:4239 writesame 0x98ee636 thru 0x98f9c28 data_size 0x1(0xb5f3 bytes) 2024-09-16T13:18:01.625 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:01.625 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:01.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:01.653 INFO:teuthology.orchestra.run.smithi098.stdout:4240 trunc from 0xaed89eb to 0x716acc2 2024-09-16T13:18:01.654 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:01.644+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:01.702 INFO:teuthology.orchestra.run.smithi098.stdout:4241 write 0xdf37d04 thru 0xdf3b694 (0x3991 bytes) 2024-09-16T13:18:01.712 INFO:teuthology.orchestra.run.smithi098.stdout:4242 read 0xac2b477 thru 0xac34a90 (0x961a bytes) 2024-09-16T13:18:01.717 INFO:teuthology.orchestra.run.smithi098.stdout:4243 trunc from 0xdf3b695 to 0xb858dac 2024-09-16T13:18:01.738 INFO:teuthology.orchestra.run.smithi098.stdout:4244 writesame 0x2f046bc thru 0x2f092c7 data_size 0x3e(0x4c0c bytes) 2024-09-16T13:18:01.744 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:01.744 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:01.746 INFO:teuthology.orchestra.run.smithi098.stdout:4246 read 0x29205b2 thru 0x2925019 (0x4a68 bytes) 2024-09-16T13:18:01.746 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:01.746 INFO:teuthology.orchestra.run.smithi098.stdout:4248 write 0x50c39b4 thru 0x50cd4b2 (0x9aff bytes) 2024-09-16T13:18:01.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:01.755 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:01.747+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:18:01.772 INFO:teuthology.orchestra.run.smithi098.stdout:4250 write 0xd8c3bab thru 0xd8d2de5 (0xf23b bytes) 2024-09-16T13:18:01.784 INFO:teuthology.orchestra.run.smithi098.stdout:4253 read 0xcc47e16 thru 0xcc567dc (0xe9c7 bytes) 2024-09-16T13:18:01.784 INFO:teuthology.orchestra.run.smithi098.stdout:4254 writesame 0xa530535 thru 0xa539da6 data_size 0x4f(0x9872 bytes) 2024-09-16T13:18:01.792 INFO:teuthology.orchestra.run.smithi098.stdout:4255 read 0x489ca22 thru 0x48a4377 (0x7956 bytes) 2024-09-16T13:18:01.809 INFO:teuthology.orchestra.run.smithi098.stdout:4256 trunc from 0xd8d2de6 to 0x642156e 2024-09-16T13:18:01.832 INFO:teuthology.orchestra.run.smithi098.stdout:4257 read 0x4166c68 thru 0x416d12f (0x64c8 bytes) 2024-09-16T13:18:01.846 INFO:teuthology.orchestra.run.smithi098.stdout:4258 writesame 0x7e45ba9 thru 0x7e4d71c data_size 0x2(0x7b74 bytes) 2024-09-16T13:18:01.856 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:01.848+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:01.860 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:01.861 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:01.864 INFO:teuthology.orchestra.run.smithi098.stdout:4259 write 0xd08ea7f thru 0xd09d9fa (0xef7c bytes) 2024-09-16T13:18:01.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:01.877 INFO:teuthology.orchestra.run.smithi098.stdout:4260 trunc from 0xd09d9fb to 0xd1bb2f8 2024-09-16T13:18:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:4261 read 0x86aeeab thru 0x86bbc95 (0xcdeb bytes) 2024-09-16T13:18:01.887 INFO:teuthology.orchestra.run.smithi098.stdout:4262 write 0xc853b24 thru 0xc8592ea (0x57c7 bytes) 2024-09-16T13:18:01.915 INFO:teuthology.orchestra.run.smithi098.stdout:4264 trunc from 0xd1bb2f8 to 0x46b0203 2024-09-16T13:18:01.946 INFO:teuthology.orchestra.run.smithi098.stdout:4265 trunc from 0x46b0203 to 0x9e37cfe 2024-09-16T13:18:01.957 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:01.951+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:18:01.961 INFO:teuthology.orchestra.run.smithi098.stdout:4266 writesame 0x753c9aa thru 0x753f66d data_size 0x1e(0x2cc4 bytes) 2024-09-16T13:18:01.967 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:01.967 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:01.967 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:01.967 INFO:teuthology.orchestra.run.smithi098.stdout:4270 punch from 0x4d0b4a2 to 0x4d17971, (0xc4cf bytes) 2024-09-16T13:18:01.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:18:01.983 INFO:teuthology.orchestra.run.smithi098.stdout:4271 writesame 0xbe4e4a0 thru 0xbe4f2a5 data_size 0x5(0xe06 bytes) 2024-09-16T13:18:01.999 INFO:teuthology.orchestra.run.smithi098.stdout:4272 trunc from 0xbe4f2a6 to 0x45a07b4 2024-09-16T13:18:02.033 INFO:teuthology.orchestra.run.smithi098.stdout:4274 write 0x4aa3194 thru 0x4aa8f74 (0x5de1 bytes) 2024-09-16T13:18:02.043 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:02.043 INFO:teuthology.orchestra.run.smithi098.stdout:4279 read 0x4184eba thru 0x4192b7d (0xdcc4 bytes) 2024-09-16T13:18:02.043 INFO:teuthology.orchestra.run.smithi098.stdout:4281 read 0x1066324 thru 0x106bc30 (0x590d bytes) 2024-09-16T13:18:02.043 INFO:teuthology.orchestra.run.smithi098.stdout:4283 write 0x1639e2a thru 0x163f130 (0x5307 bytes) 2024-09-16T13:18:02.055 INFO:teuthology.orchestra.run.smithi098.stdout:4284 trunc from 0x4aa8f75 to 0x6dae3de 2024-09-16T13:18:02.057 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:02.048+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:02.071 INFO:teuthology.orchestra.run.smithi098.stdout:4286 read 0x50760bb thru 0x5079d16 (0x3c5c bytes) 2024-09-16T13:18:02.071 INFO:teuthology.orchestra.run.smithi098.stdout:4287 read 0x33c7812 thru 0x33d29f8 (0xb1e7 bytes) 2024-09-16T13:18:02.071 INFO:teuthology.orchestra.run.smithi098.stdout:4288 write 0x3a2b80f thru 0x3a2ff57 (0x4749 bytes) 2024-09-16T13:18:02.079 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:02.079 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:02.079 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:02.079 INFO:teuthology.orchestra.run.smithi098.stdout:4290 read 0x2c0e374 thru 0x2c13fa7 (0x5c34 bytes) 2024-09-16T13:18:02.080 INFO:teuthology.orchestra.run.smithi098.stdout:4292 trunc from 0x6dae3de to 0x75746a5 2024-09-16T13:18:02.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:18:02.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:18:02.121 INFO:teuthology.orchestra.run.smithi098.stdout:4293 writesame 0xe066fb6 thru 0xe07373f data_size 0x1(0xc78a bytes) 2024-09-16T13:18:02.121 INFO:teuthology.orchestra.run.smithi098.stdout:4294 write 0xd2caeed thru 0xd2d56d8 (0xa7ec bytes) 2024-09-16T13:18:02.131 INFO:teuthology.orchestra.run.smithi098.stdout:4296 write 0xd70778d thru 0xd70bb7d (0x43f1 bytes) 2024-09-16T13:18:02.140 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:02.140 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:02.157 INFO:teuthology.orchestra.run.smithi098.stdout:4300 trunc from 0xe073740 to 0x45b1e0d 2024-09-16T13:18:02.158 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:02.148+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:02.177 INFO:teuthology.orchestra.run.smithi098.stdout:4301 read 0x3452866 thru 0x3453876 (0x1011 bytes) 2024-09-16T13:18:02.177 INFO:teuthology.orchestra.run.smithi098.stdout:4302 writesame 0x125053f thru 0x125c857 data_size 0x7(0xc319 bytes) 2024-09-16T13:18:02.188 INFO:teuthology.orchestra.run.smithi098.stdout:4303 clone 28 order 24 su 65536 sc 5 2024-09-16T13:18:02.191 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:02.191 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:02.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:18:02.259 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:02.252+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:02.297 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:02.297 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:02.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:02.361 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:02.355+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:02.414 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:02.414 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:02.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:02.462 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:02.452+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:02.521 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:02.521 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:02.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:02.579 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:02.572+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:02.628 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:02.628 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:02.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:02.663 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:02.659+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:02.734 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:02.734 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:02.743 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:02.764 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:02.755+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:02.844 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:02.844 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:02.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:02.865 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:02.859+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:02.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:02.965 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:02.956+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:03.064 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:03.064 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:03.066 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:03.059+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:03.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:03.124 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:18:03.167 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:03.159+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:18:03.181 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:03.181 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:03.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:03.268 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:03.260+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:03.290 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:03.290 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:03.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:03.368 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:03.360+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:03.389 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone27 from 0x45b1e0d (overlap 0x9da86d) to 0x0 2024-09-16T13:18:03.411 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:03.411 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:03.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:03.469 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:03.460+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:03.518 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:03.518 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:03.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:03.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:03.570 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:03.563+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:18:03.626 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:03.626 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:03.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:03.671 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:03.664+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:03.694 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:03.715 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:03.737 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:03.747 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:03.747 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:03.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:03.759 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:03.771 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:03.763+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:03.781 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:03.796 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #26, image image_client.0-clone26 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent27 2024-09-16T13:18:03.801 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:03.833 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:03.854 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:03.856 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:03.857 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:03.857 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:03.857 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:03.857 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:03.857 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:03.857 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:03.857 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:18:03.863 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:03.863 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:03.863 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:03.863 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:03.863 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:03.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:03.873 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:03.864+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:03.882 INFO:teuthology.orchestra.run.smithi098.stdout:4304 trunc from 0x45b1e0d to 0xcb5b882 2024-09-16T13:18:03.901 INFO:teuthology.orchestra.run.smithi098.stdout:4305 read 0x761a06d thru 0x7622ddc (0x8d70 bytes) 2024-09-16T13:18:03.901 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:03.902 INFO:teuthology.orchestra.run.smithi098.stdout:4307 writesame 0x25848af thru 0x2590730 data_size 0x5(0xbe82 bytes) 2024-09-16T13:18:03.942 INFO:teuthology.orchestra.run.smithi098.stdout:4308 read 0x57d2dfa thru 0x57d74f3 (0x46fa bytes) 2024-09-16T13:18:03.942 INFO:teuthology.orchestra.run.smithi098.stdout:4311 write 0x7f58533 thru 0x7f5faa2 (0x7570 bytes) 2024-09-16T13:18:03.947 INFO:teuthology.orchestra.run.smithi098.stdout:4313 write 0x1b4d0bd thru 0x1b59be4 (0xcb28 bytes) 2024-09-16T13:18:03.973 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:03.964+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:03.980 INFO:teuthology.orchestra.run.smithi098.stdout:4314 write 0xdabdce4 thru 0xdac11dd (0x34fa bytes) 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8369 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.924946599000002, 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020058919000000001, 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:03.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8369 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:03.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056334+0000", 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056335+0000", 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056338+0000", 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000001e-06 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:03.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056344+0000", 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8150000000000002e-06 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056348+0000", 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1080000000000003e-06 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.070951+0000", 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014603055 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071060+0000", 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109462 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.071508+0000", 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044824200000000001 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074304+0000", 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027951719999999998 2024-09-16T13:18:03.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076331+0000", 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020276529999999999 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076369+0000", 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7557e-05 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:03.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076393+0000", 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4445999999999999e-05 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8370 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.924929280999997, 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021240851000000002, 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:03.994 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:03.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:03.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:03.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:03.996 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8370 2024-09-16T13:18:03.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.996 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:03.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056352+0000", 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999999e-07 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056355+0000", 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.667e-06 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056360+0000", 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6620000000000002e-06 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056366+0000", 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072672+0000", 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016306157000000002 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.072746+0000", 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3648999999999999e-05 2024-09-16T13:18:03.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.073364+0000", 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061885499999999997 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076445+0000", 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003080396 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077482+0000", 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001036709 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077535+0000", 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3211000000000003e-05 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077592+0000", 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7664000000000002e-05 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.001 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8371 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.924900485999999, 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021386721000000001, 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.002 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8371 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056380+0000", 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056381+0000", 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056385+0000", 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056391+0000", 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6959999999999999e-06 2024-09-16T13:18:04.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056396+0000", 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2549999999999999e-06 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074355+0000", 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017959098999999999 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074503+0000", 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014797200000000001 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.074955+0000", 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045213300000000002 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077607+0000", 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026519500000000001 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077660+0000", 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2911000000000002e-05 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077725+0000", 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4991000000000002e-05 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077767+0000", 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2509000000000002e-05 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.007 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8372 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.924879464999997, 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021557531000000001, 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-16T13:18:04.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056401+0000", 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056402+0000", 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1399999999999997e-07 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056405+0000", 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6520000000000002e-06 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056411+0000", 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2009999999999996e-06 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056415+0000", 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.211e-06 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075482+0000", 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019067163000000002 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075546+0000", 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3516000000000006e-05 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.075844+0000", 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029810999999999998 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077784+0000", 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019399350000000001 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077823+0000", 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9524000000000003e-05 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077917+0000", 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4090000000000002e-05 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.077959+0000", 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1615000000000001e-05 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.013 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8373 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.924859122000001, 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022897421000000001, 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3200000000000005e-07 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056427+0000", 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-06 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056433+0000", 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056437+0000", 2024-09-16T13:18:04.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076470+0000", 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020032239 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076524+0000", 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4475000000000003e-05 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076996+0000", 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000471905 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.078749+0000", 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001752539 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079217+0000", 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046819400000000001 2024-09-16T13:18:04.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079276+0000", 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9407000000000002e-05 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079319+0000", 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8398 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.895495330000003, 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017439282, 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8398 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.020 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085786+0000", 2024-09-16T13:18:04.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4700000000000002e-07 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085788+0000", 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4719999999999998e-06 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085793+0000", 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6179999999999997e-06 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085796+0000", 2024-09-16T13:18:04.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6680000000000001e-06 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099514+0000", 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013717895000000001 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.099873+0000", 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035970799999999998 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:04.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100269+0000", 2024-09-16T13:18:04.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000395382 2024-09-16T13:18:04.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103068+0000", 2024-09-16T13:18:04.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027992680000000002 2024-09-16T13:18:04.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103114+0000", 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6189000000000001e-05 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103178+0000", 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3845999999999998e-05 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103225+0000", 2024-09-16T13:18:04.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6888999999999998e-05 2024-09-16T13:18:04.026 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.026 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.026 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.026 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8399 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:04.026 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:18:04.026 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.895453852999999, 2024-09-16T13:18:04.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017551573000000001, 2024-09-16T13:18:04.026 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8399 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085827+0000", 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085828+0000", 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085830+0000", 2024-09-16T13:18:04.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8609999999999998e-06 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085835+0000", 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.437e-06 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085837+0000", 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.571e-06 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.100950+0000", 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015112537000000001 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101047+0000", 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6945000000000001e-05 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101468+0000", 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000420794 2024-09-16T13:18:04.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103315+0000", 2024-09-16T13:18:04.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018474769999999999 2024-09-16T13:18:04.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103321+0000", 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103356+0000", 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4764000000000001e-05 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103379+0000", 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3102999999999999e-05 2024-09-16T13:18:04.032 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8400 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.895440952000001, 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017960752, 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.033 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8400 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085840+0000", 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.53e-07 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085842+0000", 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2620000000000001e-06 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085847+0000", 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2119999999999997e-06 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.085849+0000", 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.204e-06 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.101995+0000", 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016146384999999999 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102100+0000", 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010482199999999999 2024-09-16T13:18:04.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.102466+0000", 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000365731 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103389+0000", 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092367199999999995 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103702+0000", 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031220400000000001 2024-09-16T13:18:04.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103754+0000", 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2318999999999998e-05 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.103801+0000", 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6687999999999998e-05 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8539 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.737309396999997, 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018486425000000001, 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8539 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243972+0000", 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0500000000000003e-07 2024-09-16T13:18:04.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243975+0000", 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800000000000002e-06 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243982+0000", 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243986+0000", 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4100000000000001e-06 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258523+0000", 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014536604999999999 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.258678+0000", 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000155341 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.259218+0000", 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053930600000000003 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262325+0000", 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031076799999999998 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262367+0000", 2024-09-16T13:18:04.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1480000000000003e-05 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262417+0000", 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0555e-05 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262458+0000", 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0556999999999998e-05 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8540 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.737289976, 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018906683000000001, 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8540 2024-09-16T13:18:04.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.046 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:18:04.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243991+0000", 2024-09-16T13:18:04.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243992+0000", 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.75e-07 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.243995+0000", 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1219999999999999e-06 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244001+0000", 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5010000000000001e-06 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244006+0000", 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999996e-06 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260137+0000", 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016131 2024-09-16T13:18:04.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260200+0000", 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3782999999999996e-05 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.260598+0000", 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039746600000000002 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262331+0000", 2024-09-16T13:18:04.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017327849999999999 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262786+0000", 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045546300000000003 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262851+0000", 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-05 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.262898+0000", 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7098000000000002e-05 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8541 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.737270752999997, 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019366889000000002, 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.053 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8541 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244010+0000", 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244011+0000", 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244014+0000", 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.79e-06 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244020+0000", 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6370000000000002e-06 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.244025+0000", 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3359999999999997e-06 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261122+0000", 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017097161 2024-09-16T13:18:04.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261228+0000", 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105989 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.261475+0000", 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024757000000000002 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263022+0000", 2024-09-16T13:18:04.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015466760000000001 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263300+0000", 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027828100000000001 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263344+0000", 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4008999999999998e-05 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.263377+0000", 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2832e-05 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11776 3.1a 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.463867745999998, 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027432719000000001, 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11776 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517415+0000", 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5239999999999999e-06 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526328+0000", 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089138059999999998 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526362+0000", 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3775000000000001e-05 2024-09-16T13:18:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526379+0000", 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6835000000000001e-05 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526467+0000", 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7703999999999999e-05 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526653+0000", 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186379 2024-09-16T13:18:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.527624+0000", 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097080400000000002 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543313+0000", 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015689456000000001 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544696+0000", 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013822979999999999 2024-09-16T13:18:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544784+0000", 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8419999999999997e-05 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544846+0000", 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1717999999999994e-05 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:11778 3.1 e40/23 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head v 40'57, mlcod=40'55)", 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.457383718999999, 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01745787, 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523897+0000", 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.523898+0000", 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-06 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533067+0000", 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091691120000000001 2024-09-16T13:18:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533101+0000", 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3733999999999999e-05 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533115+0000", 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3410999999999999e-05 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533194+0000", 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9827000000000007e-05 2024-09-16T13:18:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.533223+0000", 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8937000000000001e-05 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541297+0000", 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080739850000000005 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.541355+0000", 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7708999999999999e-05 2024-09-16T13:18:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11777 3.1a 3:59cb7a3f:::rbd_object_map.112d21cb8818:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.454878807999997, 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018649718999999999, 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11777 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526402+0000", 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526404+0000", 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.654e-06 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526409+0000", 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.882e-06 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526424+0000", 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5328999999999998e-05 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526428+0000", 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6219999999999998e-06 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542209+0000", 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015781765999999999 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542334+0000", 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125148 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.542804+0000", 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046944899999999998 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543732+0000", 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092824699999999999 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544929+0000", 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001196522 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545004+0000", 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4962999999999994e-05 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.545052+0000", 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8136999999999997e-05 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11779 3.9 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:18:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.434357226000003, 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020806181999999999, 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11779 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546924+0000", 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.546926+0000", 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9769999999999999e-06 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552330+0000", 2024-09-16T13:18:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054043980000000004 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552353+0000", 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2875999999999999e-05 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552362+0000", 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7299999999999994e-06 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552447+0000", 2024-09-16T13:18:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5620999999999999e-05 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.552710+0000", 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026273300000000002 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2", 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.553446+0000", 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073563299999999997 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.559002+0000", 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055564580000000002 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567615+0000", 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086126680000000004 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567689+0000", 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3839999999999998e-05 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.567730+0000", 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1248000000000002e-05 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13736 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.683286049000003, 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018229967999999999, 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13736 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297995+0000", 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2e-07 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.297998+0000", 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.824e-06 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298004+0000", 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8730000000000004e-06 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298008+0000", 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312651+0000", 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014643351000000001 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.312740+0000", 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8466000000000003e-05 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313081+0000", 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034067899999999998 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315982+0000", 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002900943 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316051+0000", 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9029999999999995e-05 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316101+0000", 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0680000000000003e-05 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316225+0000", 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012359099999999999 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13737 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:18:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.683268929, 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018381208999999999, 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13737 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298012+0000", 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298015+0000", 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6869999999999999e-06 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298021+0000", 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7799999999999997e-06 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298027+0000", 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.071e-06 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.313829+0000", 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015801696 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314019+0000", 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018995700000000001 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.314379+0000", 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036036500000000002 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315988+0000", 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016084949999999999 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316290+0000", 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000302732 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316324+0000", 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3937999999999999e-05 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316393+0000", 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8966000000000003e-05 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13738 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.683249947, 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021400311000000002, 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13738 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298031+0000", 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298032+0000", 2024-09-16T13:18:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7400000000000003e-07 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298034+0000", 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5770000000000001e-06 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298040+0000", 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7010000000000001e-06 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298044+0000", 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2069999999999997e-06 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315033+0000", 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016988970999999999 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315106+0000", 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2723000000000002e-05 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.315428+0000", 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032191099999999997 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319249+0000", 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003821166 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319313+0000", 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3996000000000002e-05 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319347+0000", 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4591999999999997e-05 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319431+0000", 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3893000000000005e-05 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13739 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.683232435000001, 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021506517999999999, 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13739 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298048+0000", 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298049+0000", 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298052+0000", 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.819e-06 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298058+0000", 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6960000000000002e-06 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298061+0000", 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7560000000000001e-06 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316465+0000", 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018403744 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.316747+0000", 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028175200000000002 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.317694+0000", 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000947115 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319304+0000", 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016102029999999999 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319446+0000", 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142008 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319490+0000", 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3690999999999999e-05 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319555+0000", 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5191000000000007e-05 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13740 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.683178951999999, 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023881870999999999, 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13740 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298103+0000", 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298106+0000", 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1559999999999999e-06 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298120+0000", 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4474999999999999e-05 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298124+0000", 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1699999999999999e-06 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319635+0000", 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021510129999999999 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319762+0000", 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012746400000000001 2024-09-16T13:18:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.320225+0000", 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046264800000000002 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321629+0000", 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00140383 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321682+0000", 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3720000000000001e-05 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321757+0000", 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321984+0000", 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022684199999999999 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.107 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:04.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:04.109 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:18:04.109 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:18:04.109 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-16T13:18:04.109 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:18:04.109 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:18:04.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-16T13:18:04.110 INFO:teuthology.orchestra.run.smithi098.stdout:4315 write 0x5daa33c thru 0x5db54d9 (0xb19e bytes) 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:4318 punch from 0x86f111b to 0x86fb903, (0xa7e8 bytes) 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:4319 writesame 0x8e14800 thru 0x8e20805 data_size 0x9(0xc006 bytes) 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:4320 punch from 0x1633530 to 0x163d70e, (0xa1de bytes) 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:4321 punch from 0xc33c7e6 to 0xc33e4cd, (0x1ce7 bytes) 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:4325 writesame 0x614bd8f thru 0x615360c data_size 0x6(0x787e bytes) 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:4327 writesame 0xc0210a3 thru 0xc030cc8 data_size 0x19(0xfc26 bytes) 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:4329 write 0x1269b80 thru 0x126d6e5 (0x3b66 bytes) 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:4330 punch from 0x57ee20c to 0x57f791b, (0x970f bytes) 2024-09-16T13:18:04.111 INFO:teuthology.orchestra.run.smithi098.stdout:4331 punch from 0xe0bfdf to 0xe1b0da, (0xf0fb bytes) 2024-09-16T13:18:04.112 INFO:teuthology.orchestra.run.smithi098.stdout:4332 write 0x4ede2c7 thru 0x4ee9016 (0xad50 bytes) 2024-09-16T13:18:04.112 INFO:teuthology.orchestra.run.smithi098.stdout:4334 trunc from 0xdac11de to 0xa4a9631 2024-09-16T13:18:04.112 INFO:teuthology.orchestra.run.smithi098.stdout:4337 read 0x9fd8309 thru 0x9fd8f3b (0xc33 bytes) 2024-09-16T13:18:04.112 INFO:teuthology.orchestra.run.smithi098.stdout:4338 read 0x485bb7a thru 0x485c013 (0x49a bytes) 2024-09-16T13:18:04.112 INFO:teuthology.orchestra.run.smithi098.stdout:4340 trunc from 0xa4a9631 to 0x7264c99 2024-09-16T13:18:04.112 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:04.100+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:04.130 INFO:teuthology.orchestra.run.smithi098.stdout:4341 write 0xe778daf thru 0xe784def (0xc041 bytes) 2024-09-16T13:18:04.161 INFO:teuthology.orchestra.run.smithi098.stdout:4342 trunc from 0xe784df0 to 0x6c6057 2024-09-16T13:18:04.211 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:04.204+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:04.221 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:04.221 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:04.221 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:04.222 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:04.224 INFO:teuthology.orchestra.run.smithi098.stdout:4343 write 0x9975c83 thru 0x9981d3e (0xc0bc bytes) 2024-09-16T13:18:04.227 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:04.227 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "" 2024-09-16T13:18:04.227 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:04.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:04.232 INFO:teuthology.orchestra.run.smithi098.stdout:4344 writesame 0x407914c thru 0x407bc5f data_size 0x6(0x2b14 bytes) 2024-09-16T13:18:04.237 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:04.256 INFO:teuthology.orchestra.run.smithi098.stdout:4346 write 0xebbe8fb thru 0xebc2eea (0x45f0 bytes) 2024-09-16T13:18:04.265 INFO:teuthology.orchestra.run.smithi098.stdout:4347 clone 29 order 23 su 131072 sc 9 2024-09-16T13:18:04.311 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:04.303+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:04.335 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:04.335 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:04.335 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:04.335 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:04.336 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:04.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:04.413 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:04.407+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:18:04.491 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2524 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.461023828999998, 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014381392, 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2524 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:18:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024667+0000", 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024668+0000", 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8400000000000003e-07 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024670+0000", 2024-09-16T13:18:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.849e-06 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024675+0000", 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.451e-06 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024678+0000", 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8739999999999999e-06 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038723+0000", 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014044562999999999 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.038811+0000", 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039049+0000", 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023761100000000001 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2525 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 557056~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460989482000002, 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014602618, 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2525 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024705+0000", 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4650000000000001e-06 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024711+0000", 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024716+0000", 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1710000000000004e-06 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039096+0000", 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014380184000000001 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039187+0000", 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1065999999999999e-05 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039304+0000", 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117333 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2526 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1507328~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460971313000002, 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014823846, 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2526 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:18:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024722+0000", 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024729+0000", 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024733+0000", 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:18:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039337+0000", 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014604026000000001 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039427+0000", 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0437000000000002e-05 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039544+0000", 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116234 2024-09-16T13:18:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2527 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1605632~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460953549999999, 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015041116, 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2527 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024738+0000", 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024746+0000", 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2130000000000008e-06 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024753+0000", 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024758+0000", 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7670000000000003e-06 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014820533 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039670+0000", 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1825999999999998e-05 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039779+0000", 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108424 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2528 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460928111999998, 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015245242000000001, 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2528 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024765+0000", 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1040000000000001e-06 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024772+0000", 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024776+0000", 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039810+0000", 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015034194000000001 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039897+0000", 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6864e-05 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040008+0000", 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110802 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2529 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2555904~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460908992999997, 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015471574, 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2529 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024785+0000", 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5969999999999999e-06 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024800+0000", 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4681999999999999e-05 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024806+0000", 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040039+0000", 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015233007999999999 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040143+0000", 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010322799999999999 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040254+0000", 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011092899999999999 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2530 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460878713, 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015667606000000001, 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2530 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024813+0000", 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024815+0000", 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5160000000000001e-06 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024827+0000", 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1915e-05 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024834+0000", 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5180000000000001e-06 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040286+0000", 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015452565999999999 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040375+0000", 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040480+0000", 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105388 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2531 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3538944~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460851497999997, 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015867612999999999, 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2531 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024843+0000", 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6960000000000001e-06 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024856+0000", 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-05 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024863+0000", 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6069999999999996e-06 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040513+0000", 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650662999999999 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040599+0000", 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5439999999999995e-05 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040707+0000", 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108461 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2532 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460820992999999, 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061912000000001, 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2532 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:18:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024871+0000", 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024874+0000", 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024881+0000", 2024-09-16T13:18:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0570000000000004e-06 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024886+0000", 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040742+0000", 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015856808 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040824+0000", 2024-09-16T13:18:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040932+0000", 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108378 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2533 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:18:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460800468000002, 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016272282999999998, 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2533 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024893+0000", 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419999999999999e-06 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024900+0000", 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024903+0000", 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040965+0000", 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061565999999999 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041052+0000", 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7213000000000007e-05 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041163+0000", 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011066899999999999 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2534 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3604480~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460781937, 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016482746999999999, 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2534 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024910+0000", 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1299999999999999e-07 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024912+0000", 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024920+0000", 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024924+0000", 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.036e-06 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041197+0000", 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016273242 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041282+0000", 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5013999999999996e-05 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041392+0000", 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109567 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2535 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460761275000003, 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016691902000000002, 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2535 2024-09-16T13:18:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:18:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7599999999999997e-07 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024932+0000", 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024949+0000", 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7053000000000001e-05 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024961+0000", 2024-09-16T13:18:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041425+0000", 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016463851000000002 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041515+0000", 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9549999999999995e-05 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041622+0000", 2024-09-16T13:18:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107132 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2536 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1638400~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460723469000001, 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016880869999999999, 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2536 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024971+0000", 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.615e-06 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024988+0000", 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6912999999999999e-05 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024993+0000", 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041653+0000", 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016660245000000001 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041743+0000", 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0099999999999995e-05 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041848+0000", 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105404 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2537 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1671168~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460692715999997, 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017075346000000002, 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2537 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024999+0000", 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025002+0000", 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025009+0000", 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025013+0000", 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1840000000000001e-06 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041886+0000", 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016873249999999999 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041971+0000", 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4487000000000006e-05 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042074+0000", 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102847 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2538 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2621440~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460672545000001, 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017314057000000001, 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2538 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025019+0000", 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025021+0000", 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025033+0000", 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1985e-05 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025043+0000", 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3810000000000005e-06 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042107+0000", 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017064471000000001 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042203+0000", 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5810999999999995e-05 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042333+0000", 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012956099999999999 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2539 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.460644817000002, 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017500180000000001, 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2539 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025047+0000", 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025053+0000", 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025058+0000", 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025061+0000", 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.072e-06 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042365+0000", 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017304885999999998 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042458+0000", 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2991000000000005e-05 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042546+0000", 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7904000000000004e-05 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:22787 3.17 3:edd0f01e:::rbd_object_map.112db5030c67:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.168260235000002, 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015269972, 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 22787 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317432+0000", 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.127e-06 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317439+0000", 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0500000000000003e-06 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317458+0000", 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9326000000000001e-05 2024-09-16T13:18:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317467+0000", 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0669999999999998e-06 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329243+0000", 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011775551 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329293+0000", 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9709000000000001e-05 2024-09-16T13:18:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329505+0000", 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021237200000000001 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.330969+0000", 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014635290000000001 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332590+0000", 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001621485 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332655+0000", 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4670000000000003e-05 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332701+0000", 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6085999999999999e-05 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:23429 3.5 3:a41c7f85:::rbd_object_map.112dc49cabe2:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T13:18:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.396564068, 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014415534000000001, 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 23429 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089128+0000", 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7599999999999996e-07 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089140+0000", 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1939999999999999e-05 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089164+0000", 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4430000000000002e-05 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089173+0000", 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8200000000000003e-06 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.097878+0000", 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087055320000000002 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.098035+0000", 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156537 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.098605+0000", 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057031900000000001 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103250+0000", 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046448119999999999 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103351+0000", 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100763 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103472+0000", 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121091 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103542+0000", 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0513999999999996e-05 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:23430 3.5 3:a41c7f85:::rbd_object_map.112dc49cabe2:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.396508178000001, 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016212550999999999, 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 23430 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089184+0000", 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:18:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089192+0000", 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8730000000000007e-06 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089210+0000", 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8127000000000001e-05 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089217+0000", 2024-09-16T13:18:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7130000000000005e-06 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.099775+0000", 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010557957999999999 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.099894+0000", 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011883999999999999 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.100458+0000", 2024-09-16T13:18:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056368200000000003 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103330+0000", 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028723690000000001 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105122+0000", 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017912430000000001 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105297+0000", 2024-09-16T13:18:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175055 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105395+0000", 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8758000000000006e-05 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:18:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.6632197939999998, 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:18:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.580 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:04.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:04.581 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:04.572+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:04.682 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:04.675+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:04.689 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:04.689 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:04.689 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:04.689 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:04.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:04.783 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:04.775+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:04.795 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:04.795 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:04.795 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:04.795 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:04.795 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:04.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:04.884 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:04.875+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:18:04.899 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:04.899 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:04.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:04.899 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11487 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 430080~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.408440738000003, 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014286111000000001, 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11487 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.900 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486859+0000", 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486860+0000", 2024-09-16T13:18:04.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.04e-06 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486866+0000", 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0629999999999996e-06 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486881+0000", 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5023e-05 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.486889+0000", 2024-09-16T13:18:04.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8149999999999995e-06 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.500906+0000", 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014016416 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501008+0000", 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102243 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501146+0000", 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013751099999999999 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11535 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.407757764999999, 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014337727999999999, 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11535 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487542+0000", 2024-09-16T13:18:04.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487543+0000", 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023e-06 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487551+0000", 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0309999999999998e-06 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487566+0000", 2024-09-16T13:18:04.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5000999999999999e-05 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487578+0000", 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1066e-05 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501655+0000", 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014077151 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501747+0000", 2024-09-16T13:18:04.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2670999999999994e-05 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501880+0000", 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013278500000000001 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11541 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 413696~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.407713215000001, 2024-09-16T13:18:04.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014490381, 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11541 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487587+0000", 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487588+0000", 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487592+0000", 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4920000000000004e-06 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487607+0000", 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4926000000000001e-05 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487616+0000", 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6829999999999997e-06 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501906+0000", 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0142898 2024-09-16T13:18:04.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.501959+0000", 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2985000000000002e-05 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502077+0000", 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118495 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11542 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 393216~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.40767469, 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014676434, 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11542 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487625+0000", 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487626+0000", 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.034e-06 2024-09-16T13:18:04.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487631+0000", 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7659999999999998e-06 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487646+0000", 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4786e-05 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487655+0000", 2024-09-16T13:18:04.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6489999999999997e-06 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502110+0000", 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014455749 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502209+0000", 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8608000000000002e-05 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502302+0000", 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2842000000000003e-05 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11544 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 466944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.407633728999997, 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014846796000000001, 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11544 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487666+0000", 2024-09-16T13:18:04.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487667+0000", 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.569999999999999e-07 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487672+0000", 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2459999999999997e-06 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487686+0000", 2024-09-16T13:18:04.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4847e-05 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487695+0000", 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9109999999999999e-06 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502337+0000", 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014641861000000001 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502440+0000", 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010282599999999999 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502513+0000", 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3147999999999998e-05 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11547 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 339968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.921 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.407594602000003, 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015113645, 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11547 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487706+0000", 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487707+0000", 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.015e-06 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487711+0000", 2024-09-16T13:18:04.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8289999999999998e-06 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487727+0000", 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5497000000000001e-05 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487736+0000", 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1409999999999994e-06 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502557+0000", 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014821212 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502675+0000", 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117741 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502819+0000", 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014421 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11549 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 413696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.407554179999998, 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016244525999999999, 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11549 2024-09-16T13:18:04.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487746+0000", 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487747+0000", 2024-09-16T13:18:04.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.094e-06 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487752+0000", 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.758e-06 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487770+0000", 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8502999999999999e-05 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487779+0000", 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1579999999999994e-06 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502868+0000", 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015088811000000001 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.502916+0000", 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8109000000000003e-05 2024-09-16T13:18:04.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503990+0000", 2024-09-16T13:18:04.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010740929999999999 2024-09-16T13:18:04.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.930 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.930 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11578 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 409600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.407503738000003, 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015410814, 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11578 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487796+0000", 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487798+0000", 2024-09-16T13:18:04.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.646e-06 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487804+0000", 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487821+0000", 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7677999999999999e-05 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487830+0000", 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9090000000000006e-06 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503033+0000", 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015202337999999999 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503123+0000", 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0414000000000005e-05 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503207+0000", 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4172000000000007e-05 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.935 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11580 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.407460102000002, 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015544980999999999, 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11580 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487840+0000", 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487841+0000", 2024-09-16T13:18:04.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487846+0000", 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3919999999999996e-06 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487861+0000", 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4946000000000001e-05 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.487869+0000", 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8750000000000006e-06 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503240+0000", 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015370511999999999 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503334+0000", 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4265000000000006e-05 2024-09-16T13:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503385+0000", 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0773999999999999e-05 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11617 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 430080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.406140403999999, 2024-09-16T13:18:04.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376203000000001, 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11617 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489160+0000", 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489162+0000", 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3089999999999998e-06 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489171+0000", 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3430000000000002e-06 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489198+0000", 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6449999999999999e-05 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489216+0000", 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8057999999999999e-05 2024-09-16T13:18:04.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503402+0000", 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014186196999999999 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503454+0000", 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1989e-05 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503536+0000", 2024-09-16T13:18:04.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1856999999999999e-05 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11624 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 503808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.405937315999999, 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014376676, 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.945 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11624 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489363+0000", 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489365+0000", 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.795e-06 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489372+0000", 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4030000000000001e-06 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489392+0000", 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0129e-05 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489404+0000", 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1555e-05 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503558+0000", 2024-09-16T13:18:04.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014154771999999999 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503644+0000", 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5691000000000003e-05 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503740+0000", 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5330999999999999e-05 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.949 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11630 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.405883979000002, 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014893777, 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.950 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11630 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489416+0000", 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489417+0000", 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6800000000000009e-07 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489435+0000", 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7818000000000001e-05 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489450+0000", 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5072000000000001e-05 2024-09-16T13:18:04.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489459+0000", 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3009999999999995e-06 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503787+0000", 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01432719 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.503889+0000", 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102604 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504310+0000", 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042082399999999998 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.954 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11632 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.405830866000002, 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01524066, 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11632 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489469+0000", 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489470+0000", 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.78e-07 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489475+0000", 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9400000000000001e-06 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489488+0000", 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3176999999999999e-05 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489497+0000", 2024-09-16T13:18:04.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4919999999999993e-06 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504364+0000", 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014866846 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504485+0000", 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012142999999999999 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504710+0000", 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000224897 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11639 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.405789095000003, 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015375941000000001, 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.959 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11639 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489511+0000", 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489512+0000", 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4799999999999997e-07 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489517+0000", 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.938e-06 2024-09-16T13:18:04.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489532+0000", 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.47e-05 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489540+0000", 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2949999999999994e-06 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504765+0000", 2024-09-16T13:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015225449 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504832+0000", 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6929999999999998e-05 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504887+0000", 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4681e-05 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.963 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11641 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 487424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.405742584999999, 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015522312999999999, 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.964 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11641 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489558+0000", 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489559+0000", 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.066e-06 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489564+0000", 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9370000000000003e-06 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489579+0000", 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5466000000000001e-05 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489588+0000", 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9290000000000004e-06 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.504925+0000", 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015337353 2024-09-16T13:18:04.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505025+0000", 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9770000000000002e-05 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505080+0000", 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4792e-05 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11667 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 16384~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.405381816000002, 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015353510000000001, 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11667 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.969 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489918+0000", 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489920+0000", 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.84e-06 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489928+0000", 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6969999999999993e-06 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489953+0000", 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5154999999999999e-05 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489965+0000", 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2292e-05 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505115+0000", 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015149793999999999 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505184+0000", 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8858999999999999e-05 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505272+0000", 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7873000000000004e-05 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11672 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 36864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.405321411000003, 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015615053, 2024-09-16T13:18:04.973 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11672 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489979+0000", 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489980+0000", 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.217e-06 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.489985+0000", 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490000+0000", 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5603999999999998e-05 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490010+0000", 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.183e-06 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505335+0000", 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015325167000000001 2024-09-16T13:18:04.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505471+0000", 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013581699999999999 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505594+0000", 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000123168 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11676 3.10 3:0b4a9508:::rbd_data.112de4fe95f7.0000000000000001:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.405280667, 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015779395000000002, 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11676 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490019+0000", 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490021+0000", 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.575e-06 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490026+0000", 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5170000000000001e-06 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490041+0000", 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4955e-05 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490049+0000", 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5520000000000005e-06 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505647+0000", 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015597489000000001 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505743+0000", 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6908000000000001e-05 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505799+0000", 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5399000000000003e-05 2024-09-16T13:18:04.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11678 3.10 3:0b113e7d:::rbd_data.112de4fe95f7.0000000000000003:a [read 507904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.405241394000001, 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015992643000000001, 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11678 2024-09-16T13:18:04.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490059+0000", 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490060+0000", 2024-09-16T13:18:04.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3400000000000001e-06 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490065+0000", 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4830000000000001e-06 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490079+0000", 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4854999999999999e-05 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490088+0000", 2024-09-16T13:18:04.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1119999999999996e-06 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505849+0000", 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015761718000000001 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.505966+0000", 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011645399999999999 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506051+0000", 2024-09-16T13:18:04.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5680999999999995e-05 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11707 3.10 3:0bd7ebcf:::rbd_data.112de4fe95f7.0000000000000019:a [read 53248~20480] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.405022264999999, 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016106869999999999, 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:04.986 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11707 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490278+0000", 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:04.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490280+0000", 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7659999999999999e-06 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490287+0000", 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.503e-06 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490307+0000", 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9989000000000001e-05 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:04.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.490318+0000", 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0967e-05 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506171+0000", 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015853367 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506265+0000", 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3190999999999996e-05 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:04.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:04.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.506385+0000", 2024-09-16T13:18:04.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120087 2024-09-16T13:18:04.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.990 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:04.990 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:04.990 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:04.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:18:04.992 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:04.987+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:18:05.092 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:05.088+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:05.098 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:05.098 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:05.099 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:05.099 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:05.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:18:05.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:18:05.193 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:05.187+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:05.212 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:05.212 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:05.212 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:05.212 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:05.212 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:05.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:18:05.294 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:05.287+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:05.326 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:05.326 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:05.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:05.326 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:05.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.326 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:18:05.326 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:18:05.326 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 90.294588996000002, 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017053136999999999, 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025386+0000", 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025387+0000", 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025506+0000", 2024-09-16T13:18:05.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011947699999999999 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025518+0000", 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620000000000001e-05 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025525+0000", 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.593e-06 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037470+0000", 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011944742 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037598+0000", 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012856 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.037730+0000", 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131693 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.040402+0000", 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026713359999999998 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042306+0000", 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019047420000000001 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042369+0000", 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2544000000000003e-05 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042439+0000", 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9896999999999998e-05 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 90.294441590000005, 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020158395999999999, 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025533+0000", 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025534+0000", 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6000000000000002e-07 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025540+0000", 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1430000000000003e-06 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025549+0000", 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6640000000000004e-06 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025554+0000", 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1329999999999998e-06 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038446+0000", 2024-09-16T13:18:05.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012891865000000001 2024-09-16T13:18:05.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038561+0000", 2024-09-16T13:18:05.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011492 2024-09-16T13:18:05.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.038737+0000", 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017672799999999999 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.042449+0000", 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037120510000000001 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045612+0000", 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031623229999999999 2024-09-16T13:18:05.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045658+0000", 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6375999999999998e-05 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.045691+0000", 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3333000000000001e-05 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.338 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 90.294414247000006, 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020565769000000001, 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 52 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:18:05.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025560+0000", 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025561+0000", 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025567+0000", 2024-09-16T13:18:05.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7230000000000001e-06 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025575+0000", 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3590000000000008e-06 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.025581+0000", 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039699+0000", 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014117955 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039808+0000", 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109257 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.039972+0000", 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016436399999999999 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.044724+0000", 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047513540000000002 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046056+0000", 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001332471 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046098+0000", 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2318999999999999e-05 2024-09-16T13:18:05.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.046126+0000", 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7696e-05 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 89.735738984999998, 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016654710999999999, 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.344 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 74 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584236+0000", 2024-09-16T13:18:05.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584237+0000", 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584245+0000", 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584269+0000", 2024-09-16T13:18:05.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3736000000000001e-05 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584278+0000", 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7629999999999995e-06 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597476+0000", 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013197189 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597528+0000", 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2213999999999998e-05 2024-09-16T13:18:05.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597606+0000", 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7925999999999999e-05 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599728+0000", 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021220990000000001 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600806+0000", 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010786599999999999 2024-09-16T13:18:05.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600855+0000", 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600890+0000", 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5068999999999999e-05 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 89.735651392999998, 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016725059, 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584323+0000", 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584325+0000", 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584439+0000", 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114144 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584460+0000", 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0910000000000001e-05 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584477+0000", 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7323e-05 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.597953+0000", 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013475576 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598054+0000", 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101751 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.598185+0000", 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013075900000000001 2024-09-16T13:18:05.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599960+0000", 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017746299999999999 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600990+0000", 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010303160000000001 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601022+0000", 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1415000000000003e-05 2024-09-16T13:18:05.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601048+0000", 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6628999999999999e-05 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4207.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 89.735481471, 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017295360999999999, 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.355 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4207", 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/684825360", 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584493+0000", 2024-09-16T13:18:05.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584495+0000", 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1469999999999999e-06 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584504+0000", 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999992e-06 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584540+0000", 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6096000000000001e-05 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.584550+0000", 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0152999999999999e-05 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599190+0000", 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014640235 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599244+0000", 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0", 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.599320+0000", 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6369e-05 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.600646+0000", 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013251280000000001 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601714+0000", 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010680430000000001 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601759+0000", 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5107e-05 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:16:35.601788+0000", 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9768999999999999e-05 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.360 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2259 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3014656~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584152895000003, 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016650893999999999, 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2259 2024-09-16T13:18:05.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735822+0000", 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2699999999999999e-07 2024-09-16T13:18:05.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735831+0000", 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9530000000000005e-06 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735836+0000", 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7450000000000003e-06 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735839+0000", 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.915e-06 2024-09-16T13:18:05.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752296+0000", 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016457241000000001 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752360+0000", 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4108000000000004e-05 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752473+0000", 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112305 2024-09-16T13:18:05.364 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2260 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 98304~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584132973999999, 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016799073000000001, 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.365 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2260 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735842+0000", 2024-09-16T13:18:05.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735843+0000", 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3480000000000001e-06 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735847+0000", 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5990000000000002e-06 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735849+0000", 2024-09-16T13:18:05.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5859999999999999e-06 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752509+0000", 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016659377 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752559+0000", 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9774000000000002e-05 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752641+0000", 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2057000000000004e-05 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2261 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 0~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584120785000003, 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016960698, 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.369 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2261 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:18:05.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735854+0000", 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.03e-07 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735855+0000", 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735860+0000", 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1110000000000001e-06 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735862+0000", 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752695+0000", 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01683343 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752744+0000", 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8414000000000001e-05 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752814+0000", 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0623000000000005e-05 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.373 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2262 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1048576~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584110170999999, 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017224107999999998, 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2262 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735864+0000", 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735865+0000", 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.91e-07 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735866+0000", 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1480000000000001e-06 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735870+0000", 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735877+0000", 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4370000000000001e-06 2024-09-16T13:18:05.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752871+0000", 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016994061000000001 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.752962+0000", 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0470000000000006e-05 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753089+0000", 2024-09-16T13:18:05.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126642 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2263 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2031616~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584094712000002, 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017392088999999999, 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2263 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735880+0000", 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-07 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735881+0000", 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735885+0000", 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4180000000000001e-06 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735887+0000", 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-06 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753116+0000", 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017228963 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753184+0000", 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7374999999999998e-05 2024-09-16T13:18:05.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753272+0000", 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8323000000000001e-05 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2264 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2064384~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584084861000001, 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017630541999999999, 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.382 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2264 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:18:05.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735890+0000", 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735891+0000", 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.19e-06 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735895+0000", 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2370000000000001e-06 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735897+0000", 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4669999999999998e-06 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753316+0000", 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017418685 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753403+0000", 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7590000000000007e-05 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753520+0000", 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116863 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.386 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2265 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1081344~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584074979, 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017739263000000002, 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2265 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:18:05.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735900+0000", 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735901+0000", 2024-09-16T13:18:05.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735904+0000", 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.304e-06 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735907+0000", 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3599999999999999e-06 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753534+0000", 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017627446000000001 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753576+0000", 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1474999999999998e-05 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753639+0000", 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3188999999999994e-05 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.390 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2266 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 2097152~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584065746, 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017970893000000002, 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2266 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.391 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735909+0000", 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:18:05.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735914+0000", 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1699999999999996e-06 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735918+0000", 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6859999999999999e-06 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735920+0000", 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4870000000000001e-06 2024-09-16T13:18:05.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753685+0000", 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017764582000000001 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753771+0000", 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6259000000000002e-05 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753880+0000", 2024-09-16T13:18:05.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010844300000000001 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2267 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 131072~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584051844999998, 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018110804000000001, 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2267 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735923+0000", 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8700000000000002e-07 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735924+0000", 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2500000000000001e-06 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735928+0000", 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.416e-06 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735930+0000", 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4320000000000002e-06 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753898+0000", 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017967852999999999 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.753955+0000", 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6662000000000001e-05 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754034+0000", 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8904000000000002e-05 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2268 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3080192~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584042359999998, 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018252866, 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.399 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2268 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:18:05.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735932+0000", 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6600000000000003e-07 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735934+0000", 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0950000000000001e-06 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735938+0000", 2024-09-16T13:18:05.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9709999999999998e-06 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735941+0000", 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8750000000000002e-06 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754071+0000", 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018129941 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754120+0000", 2024-09-16T13:18:05.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8359000000000002e-05 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754185+0000", 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5358999999999995e-05 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2269 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584030980999998, 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018533516, 2024-09-16T13:18:05.403 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2269 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735944+0000", 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7300000000000002e-07 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735945+0000", 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1149999999999999e-06 2024-09-16T13:18:05.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735948+0000", 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5250000000000001e-06 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735951+0000", 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9909999999999998e-06 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754270+0000", 2024-09-16T13:18:05.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018318081999999999 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754325+0000", 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5648e-05 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754477+0000", 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015188199999999999 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2270 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1114112~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.584020512000002, 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019103282999999999, 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2270 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.408 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735954+0000", 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735955+0000", 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735956+0000", 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.096e-06 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735959+0000", 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5140000000000002e-06 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735963+0000", 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.053e-06 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754822+0000", 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018858883 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.754941+0000", 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118789 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755057+0000", 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116272 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2271 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 1146880~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.583998700000002, 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019332309999999998, 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.412 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2271 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:18:05.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735976+0000", 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3500000000000002e-07 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735982+0000", 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735987+0000", 2024-09-16T13:18:05.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1930000000000003e-06 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735993+0000", 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4659999999999996e-06 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755081+0000", 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019088187 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755194+0000", 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011223299999999999 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755308+0000", 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114663 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.416 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2272 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:4 [read 163840~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.583976870000001, 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019476291, 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2272 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.735998+0000", 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0999999999999999e-07 2024-09-16T13:18:05.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736003+0000", 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2279999999999998e-06 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736007+0000", 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.506e-06 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.736011+0000", 2024-09-16T13:18:05.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9280000000000003e-06 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755336+0000", 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019325189999999999 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755393+0000", 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7293999999999997e-05 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:03.755474+0000", 2024-09-16T13:18:05.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0635000000000003e-05 2024-09-16T13:18:05.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.421 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:05.421 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:05.421 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:05.422 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone28 from 0xebc2eeb (overlap 0x6c6057) to 0x6189e7 2024-09-16T13:18:05.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:05.423 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:05.412+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:05.515 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:05.515 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:05.515 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:05.515 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:05.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:05.524 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:05.515+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:05.617 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:05.618 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:05.618 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:05.618 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:05.618 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:05.625 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:05.615+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:05.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:05.671 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #27, image image_client.0-clone27 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent28 2024-09-16T13:18:05.675 INFO:teuthology.orchestra.run.smithi098.stdout:4350 writesame 0xdf0bdc9 thru 0xdf11d92 data_size 0x43(0x5fca bytes) 2024-09-16T13:18:05.694 INFO:teuthology.orchestra.run.smithi098.stdout:4351 read 0x351743f thru 0x3520d64 (0x9926 bytes) 2024-09-16T13:18:05.696 INFO:teuthology.orchestra.run.smithi098.stdout:4352 write 0x946b822 thru 0x9479ba8 (0xe387 bytes) 2024-09-16T13:18:05.710 INFO:teuthology.orchestra.run.smithi098.stdout:4354 writesame 0x32a14ba thru 0x32a88bf data_size 0x1(0x7406 bytes) 2024-09-16T13:18:05.727 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:05.720+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:05.733 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:05.733 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:05.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:05.733 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11500 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 356352~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.232521597999998, 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014537983000000001, 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11500 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496472+0000", 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496474+0000", 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.801e-06 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496480+0000", 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1140000000000002e-06 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496484+0000", 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8770000000000003e-06 2024-09-16T13:18:05.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507573+0000", 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011088473 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.507624+0000", 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1591999999999997e-05 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511010+0000", 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033857380000000001 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11501 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 430080~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.232505230999998, 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014726846, 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11501 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496488+0000", 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496489+0000", 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3700000000000001e-07 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496491+0000", 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9570000000000001e-06 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496497+0000", 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2550000000000003e-06 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.496501+0000", 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8639999999999998e-06 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511060+0000", 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014559462 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511125+0000", 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4399999999999993e-05 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511215+0000", 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.0470999999999995e-05 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11530 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 339968~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.231571043000002, 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014141661, 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.742 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11530 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497422+0000", 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497423+0000", 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-07 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497434+0000", 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1195e-05 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497442+0000", 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.323e-06 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497446+0000", 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1049999999999997e-06 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511275+0000", 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013828872000000001 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511389+0000", 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114493 2024-09-16T13:18:05.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511564+0000", 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017508700000000001 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11533 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 413696~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.231541958999998, 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014296237999999999, 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11533 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497452+0000", 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.51e-07 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497458+0000", 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9139999999999996e-06 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497464+0000", 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0469999999999999e-06 2024-09-16T13:18:05.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497467+0000", 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5030000000000002e-06 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511593+0000", 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014125692 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511669+0000", 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5950000000000003e-05 2024-09-16T13:18:05.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511748+0000", 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8681000000000001e-05 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11551 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 335872~20480] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.231485931999998, 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014415314, 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.750 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11551 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497508+0000", 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497510+0000", 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.012e-06 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497516+0000", 2024-09-16T13:18:05.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7880000000000003e-06 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497524+0000", 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9049999999999997e-06 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511769+0000", 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245809 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511818+0000", 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8092e-05 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511923+0000", 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010532299999999999 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11553 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 413696~16384] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.231463083000001, 2024-09-16T13:18:05.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014561233999999999, 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11553 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497530+0000", 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497531+0000", 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1899999999999998e-07 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497533+0000", 2024-09-16T13:18:05.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9149999999999999e-06 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497539+0000", 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.088e-06 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.497550+0000", 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0834e-05 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.511962+0000", 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411947 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512018+0000", 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6755999999999997e-05 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.512092+0000", 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3275000000000006e-05 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11637 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.228463408000003, 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013965546000000001, 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11637 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500530+0000", 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500531+0000", 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2699999999999999e-07 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500534+0000", 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.055e-06 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500542+0000", 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6389999999999998e-06 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500553+0000", 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0974e-05 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514249+0000", 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013696086 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514308+0000", 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8269000000000001e-05 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514496+0000", 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00018799600000000001 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.762 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11638 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.228433926000001, 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014058684, 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11638 2024-09-16T13:18:05.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500560+0000", 2024-09-16T13:18:05.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500563+0000", 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0790000000000001e-06 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500580+0000", 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6472000000000001e-05 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500586+0000", 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2199999999999997e-06 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514522+0000", 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013935774 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514564+0000", 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2549000000000002e-05 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.766 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514618+0000", 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3942e-05 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11642 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.228349092999999, 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014093902, 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.767 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11642 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500644+0000", 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500645+0000", 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5200000000000002e-07 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500656+0000", 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1372999999999999e-05 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500664+0000", 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9019999999999999e-06 2024-09-16T13:18:05.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500677+0000", 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2954e-05 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514649+0000", 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013971417 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514692+0000", 2024-09-16T13:18:05.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3154000000000001e-05 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514738+0000", 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6650000000000002e-05 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11646 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.228310831000002, 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014245793, 2024-09-16T13:18:05.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11646 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500683+0000", 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500685+0000", 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2019999999999998e-06 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500698+0000", 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.279e-05 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500702+0000", 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7629999999999998e-06 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514772+0000", 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014070384999999999 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514865+0000", 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2980999999999996e-05 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514928+0000", 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3114000000000006e-05 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.775 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11648 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 487424~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.228287700999999, 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014411604999999999, 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11648 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:18:05.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500706+0000", 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9900000000000001e-07 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500713+0000", 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.793e-06 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500719+0000", 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5470000000000004e-06 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.500722+0000", 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3809999999999999e-06 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.514973+0000", 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014250924 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515031+0000", 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7864e-05 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515117+0000", 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6697e-05 2024-09-16T13:18:05.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11654 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 466944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.227920867000002, 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014438998, 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11654 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501073+0000", 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8699999999999995e-07 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501075+0000", 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1940000000000001e-06 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501082+0000", 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.844e-06 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501086+0000", 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7749999999999999e-06 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515152+0000", 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014065891000000001 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515232+0000", 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0461000000000001e-05 2024-09-16T13:18:05.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515512+0000", 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00027934599999999998 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11675 3.1 3:81ec9812:::rbd_data.112de4fe95f7.0000000000000000:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.227458273000003, 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014239586, 2024-09-16T13:18:05.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11675 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501535+0000", 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501536+0000", 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8500000000000001e-07 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501539+0000", 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3529999999999999e-06 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501546+0000", 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.109e-06 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501551+0000", 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5109999999999997e-06 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515586+0000", 2024-09-16T13:18:05.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014035334999999999 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515689+0000", 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010244200000000001 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515775+0000", 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6150999999999996e-05 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11679 3.1 3:851ca016:::rbd_data.112de4fe95f7.0000000000000004:a [read 507904~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.227408349000001, 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014462922, 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11679 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501585+0000", 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501586+0000", 2024-09-16T13:18:05.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501588+0000", 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3760000000000002e-06 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501594+0000", 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9950000000000002e-06 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501598+0000", 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7239999999999998e-06 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515840+0000", 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014242546999999999 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.515949+0000", 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108591 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516048+0000", 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.9123000000000006e-05 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11686 3.1 3:878375a0:::rbd_data.112de4fe95f7.0000000000000008:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.227391777000001, 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014648357000000001, 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11686 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501602+0000", 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501604+0000", 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.846e-06 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501609+0000", 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3630000000000002e-06 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501613+0000", 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3749999999999999e-06 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516077+0000", 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014464351 2024-09-16T13:18:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516173+0000", 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5742000000000006e-05 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516250+0000", 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7312999999999997e-05 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11689 3.1 3:825a6161:::rbd_data.112de4fe95f7.000000000000000a:a [read 503808~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.227377212999997, 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014770922000000001, 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11689 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501616+0000", 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501617+0000", 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501618+0000", 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6780000000000001e-06 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501624+0000", 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3750000000000002e-06 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.501627+0000", 2024-09-16T13:18:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5149999999999998e-06 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516279+0000", 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014651928999999999 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516336+0000", 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6724000000000001e-05 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.516387+0000", 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1334e-05 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11778 3.1 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 48.210043085000002, 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022495198000000001, 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11778 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518952+0000", 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.711e-06 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522261+0000", 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033087429999999998 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522296+0000", 2024-09-16T13:18:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5335e-05 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522306+0000", 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0074999999999999e-05 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522379+0000", 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2554999999999999e-05 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522572+0000", 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019323100000000001 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.523266+0000", 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069379000000000001 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.534086+0000", 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010820112999999999 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541312+0000", 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072264440000000003 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541394+0000", 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1710000000000002e-05 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541446+0000", 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1490999999999999e-05 2024-09-16T13:18:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:25474 3.6 3:6380c99b:::rbd_object_map.112de78841b1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:49.832331+0000", 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 15.89666231, 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014819143999999999, 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 25474 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832331+0000", 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832331+0000", 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832332+0000", 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.15e-07 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832334+0000", 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.148e-06 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832337+0000", 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.832340+0000", 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4760000000000001e-06 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.842963+0000", 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010623886000000001 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.843120+0000", 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00015636400000000001 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.843657+0000", 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000536887 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.846702+0000", 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030451300000000001 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.847054+0000", 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00035228400000000002 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.847116+0000", 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1977000000000006e-05 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.847150+0000", 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4257000000000001e-05 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:25567 3.6 3:6380c99b:::rbd_object_map.112de78841b1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 15.746281801, 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.014520869, 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 25567 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982712+0000", 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7100000000000002e-07 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982715+0000", 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0639999999999998e-06 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982736+0000", 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0421999999999999e-05 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.982743+0000", 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2509999999999997e-06 2024-09-16T13:18:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.990894+0000", 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0081508239999999992 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.991014+0000", 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011978599999999999 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 1", 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.991456+0000", 2024-09-16T13:18:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00044261099999999998 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.992802+0000", 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001345398 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.997165+0000", 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043631399999999997 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.997200+0000", 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5474e-05 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:49.997233+0000", 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2327999999999999e-05 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:33885 3.1 3:80899ff9:::rbd_data.112d94eb720f.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 7864320~1841] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:03.921705+0000", 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.807288641, 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017734315, 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 33885 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.921705+0000", 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.921705+0000", 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.921706+0000", 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.531e-06 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.922864+0000", 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0011579190000000001 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.922897+0000", 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3145999999999997e-05 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.922906+0000", 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7159999999999992e-06 2024-09-16T13:18:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.922980+0000", 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3456999999999998e-05 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.923125+0000", 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000145253 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.923446+0000", 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00032101999999999999 2024-09-16T13:18:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.927202+0000", 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037558679999999999 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.939315+0000", 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0121131 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.939397+0000", 2024-09-16T13:18:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1786999999999995e-05 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:03.939439+0000", 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2518000000000002e-05 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:05.825 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:05.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:05.827 INFO:teuthology.orchestra.run.smithi098.stdout:4355 writesame 0x3b16780 thru 0x3b227aa data_size 0x1(0xc02b bytes) 2024-09-16T13:18:05.827 INFO:teuthology.orchestra.run.smithi098.stdout:4356 read 0x224b6a9 thru 0x22584f9 (0xce51 bytes) 2024-09-16T13:18:05.828 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:05.816+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:05.845 INFO:teuthology.orchestra.run.smithi098.stdout:4357 trunc from 0xebc2eeb to 0x379fa10 2024-09-16T13:18:05.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:05.927 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:05.920+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:05.951 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:05.951 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:05.952 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:05.952 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:05.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:05.966 INFO:teuthology.orchestra.run.smithi098.stdout:4358 write 0x573d2b4 thru 0x5743235 (0x5f82 bytes) 2024-09-16T13:18:06.000 INFO:teuthology.orchestra.run.smithi098.stdout:4359 write 0xce2a541 thru 0xce2afc1 (0xa81 bytes) 2024-09-16T13:18:06.005 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:06.007 INFO:teuthology.orchestra.run.smithi098.stdout:4360 read 0xcb7b345 thru 0xcb83332 (0x7fee bytes) 2024-09-16T13:18:06.007 INFO:teuthology.orchestra.run.smithi098.stdout:4361 read 0x9dde520 thru 0x9decc9c (0xe77d bytes) 2024-09-16T13:18:06.008 INFO:teuthology.orchestra.run.smithi098.stdout:4362 writesame 0x252fc3e thru 0x2539a98 data_size 0x1(0x9e5b bytes) 2024-09-16T13:18:06.025 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:06.028 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:06.023+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:06.043 INFO:teuthology.orchestra.run.smithi098.stdout:4363 read 0x9628b7b thru 0x9633674 (0xaafa bytes) 2024-09-16T13:18:06.051 INFO:teuthology.orchestra.run.smithi098.stdout:4365 trunc from 0xce2afc2 to 0x7d91572 2024-09-16T13:18:06.053 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:06.067 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:06.067 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:06.073 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:06.075 INFO:teuthology.orchestra.run.smithi098.stdout:4369 punch from 0x3686351 to 0x3689fd7, (0x3c86 bytes) 2024-09-16T13:18:06.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:06.088 INFO:teuthology.orchestra.run.smithi098.stdout:4370 write 0x8de6a5e thru 0x8defdfe (0x93a1 bytes) 2024-09-16T13:18:06.093 INFO:teuthology.orchestra.run.smithi098.stdout:4371 write 0x50aef86 thru 0x50b1f43 (0x2fbe bytes) 2024-09-16T13:18:06.096 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:06.097 INFO:teuthology.orchestra.run.smithi098.stdout:4372 write 0x69de0da thru 0x69e5874 (0x779b bytes) 2024-09-16T13:18:06.102 INFO:teuthology.orchestra.run.smithi098.stdout:4373 read 0x3471bd4 thru 0x3476325 (0x4752 bytes) 2024-09-16T13:18:06.103 INFO:teuthology.orchestra.run.smithi098.stdout:4374 punch from 0x4ee5cca to 0x4eecde0, (0x7116 bytes) 2024-09-16T13:18:06.103 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.103 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.104 INFO:teuthology.orchestra.run.smithi098.stdout:4378 read 0x2fbdedf thru 0x2fbe54f (0x671 bytes) 2024-09-16T13:18:06.104 INFO:teuthology.orchestra.run.smithi098.stdout:4379 read 0xb5e78d thru 0xb6be8f (0xd703 bytes) 2024-09-16T13:18:06.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:06.121 INFO:teuthology.orchestra.run.smithi098.stdout:4382 write 0xb287376 thru 0xb2951f3 (0xde7e bytes) 2024-09-16T13:18:06.127 INFO:teuthology.orchestra.run.smithi098.stdout:4384 write 0x707021f thru 0x707c305 (0xc0e7 bytes) 2024-09-16T13:18:06.129 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:06.123+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:06.135 INFO:teuthology.orchestra.run.smithi098.stdout:4385 write 0x915fe28 thru 0x91628b3 (0x2a8c bytes) 2024-09-16T13:18:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:18:06.146 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:06.149 INFO:teuthology.orchestra.run.smithi098.stdout:4386 write 0x1300379 thru 0x1303664 (0x32ec bytes) 2024-09-16T13:18:06.155 INFO:teuthology.orchestra.run.smithi098.stdout:4387 punch from 0x208b821 to 0x2093d58, (0x8537 bytes) 2024-09-16T13:18:06.156 INFO:teuthology.orchestra.run.smithi098.stdout:4388 write 0x1a967a0 thru 0x1a9b404 (0x4c65 bytes) 2024-09-16T13:18:06.157 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.158 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.158 INFO:teuthology.orchestra.run.smithi098.stdout:4391 trunc from 0xb2951f4 to 0xb81b23f 2024-09-16T13:18:06.167 INFO:teuthology.orchestra.run.smithi098.stdout:4393 read 0x81c0fca thru 0x81c8482 (0x74b9 bytes) 2024-09-16T13:18:06.331 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:18:06.332 INFO:teuthology.orchestra.run.smithi098.stdout:4394 trunc from 0xb81b23f to 0x6113534 2024-09-16T13:18:06.332 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:06.332 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:06.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:06.336 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.336 INFO:teuthology.orchestra.run.smithi098.stdout:4397 read 0x3237582 thru 0x323b7d3 (0x4252 bytes) 2024-09-16T13:18:06.336 INFO:teuthology.orchestra.run.smithi098.stdout:4399 write 0x1097ebf thru 0x109cb46 (0x4c88 bytes) 2024-09-16T13:18:06.336 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.336 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.336 INFO:teuthology.orchestra.run.smithi098.stdout:4402 writesame 0x1bf01c0 thru 0x1bf43ed data_size 0x2b(0x422e bytes) 2024-09-16T13:18:06.336 INFO:teuthology.orchestra.run.smithi098.stdout:4403 trunc from 0x6113534 to 0xb620462 2024-09-16T13:18:06.336 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.337 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.337 INFO:teuthology.orchestra.run.smithi098.stdout:4406 write 0x6ae919b thru 0x6af8923 (0xf789 bytes) 2024-09-16T13:18:06.337 INFO:teuthology.orchestra.run.smithi098.stdout:4408 writesame 0x3aeef2c thru 0x3af6433 data_size 0x28(0x7508 bytes) 2024-09-16T13:18:06.337 INFO:teuthology.orchestra.run.smithi098.stdout:4409 read 0x1d2ee85 thru 0x1d3b685 (0xc801 bytes) 2024-09-16T13:18:06.337 INFO:teuthology.orchestra.run.smithi098.stdout:4410 punch from 0x1621144 to 0x1625161, (0x401d bytes) 2024-09-16T13:18:06.337 INFO:teuthology.orchestra.run.smithi098.stdout:4411 read 0xa76b030 thru 0xa76b57d (0x54e bytes) 2024-09-16T13:18:06.337 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.338 INFO:teuthology.orchestra.run.smithi098.stdout:4415 trunc from 0xb620462 to 0xa249c53 2024-09-16T13:18:06.338 INFO:teuthology.orchestra.run.smithi098.stdout:4416 write 0xde9e4ff thru 0xdeab280 (0xcd82 bytes) 2024-09-16T13:18:06.338 INFO:teuthology.orchestra.run.smithi098.stdout:4418 writesame 0x674a913 thru 0x674e355 data_size 0x13(0x3a43 bytes) 2024-09-16T13:18:06.338 INFO:teuthology.orchestra.run.smithi098.stdout:4419 write 0x1d9635f thru 0x1d97dc7 (0x1a69 bytes) 2024-09-16T13:18:06.338 INFO:teuthology.orchestra.run.smithi098.stdout:4420 trunc from 0xdeab281 to 0x5d2dffb 2024-09-16T13:18:06.339 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:06.323+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:06.361 INFO:teuthology.orchestra.run.smithi098.stdout:4421 write 0xe085955 thru 0xe08f521 (0x9bcd bytes) 2024-09-16T13:18:06.384 INFO:teuthology.orchestra.run.smithi098.stdout:4422 trunc from 0xe08f522 to 0xa60f8db 2024-09-16T13:18:06.407 INFO:teuthology.orchestra.run.smithi098.stdout:4423 trunc from 0xa60f8db to 0xcd69158 2024-09-16T13:18:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:4424 read 0x84edc14 thru 0x84f6d6f (0x915c bytes) 2024-09-16T13:18:06.415 INFO:teuthology.orchestra.run.smithi098.stdout:4425 read 0x6395510 thru 0x639dec1 (0x89b2 bytes) 2024-09-16T13:18:06.431 INFO:teuthology.orchestra.run.smithi098.stdout:4426 write 0xdc46dba thru 0xdc511f4 (0xa43b bytes) 2024-09-16T13:18:06.435 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:06.424+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:06.438 INFO:teuthology.orchestra.run.smithi098.stdout:4427 write 0x864d1a thru 0x869615 (0x48fc bytes) 2024-09-16T13:18:06.442 INFO:teuthology.orchestra.run.smithi098.stdout:4428 write 0x47f5533 thru 0x47f7e97 (0x2965 bytes) 2024-09-16T13:18:06.445 INFO:teuthology.orchestra.run.smithi098.stdout:4429 read 0x1fb7bb6 thru 0x1fc477c (0xcbc7 bytes) 2024-09-16T13:18:06.446 INFO:teuthology.orchestra.run.smithi098.stdout:4430 writesame 0x79747db thru 0x797d4fe data_size 0x6(0x8d24 bytes) 2024-09-16T13:18:06.454 INFO:teuthology.orchestra.run.smithi098.stdout:4431 punch from 0x3262e48 to 0x3266233, (0x33eb bytes) 2024-09-16T13:18:06.458 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:06.458 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:06.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:06.469 INFO:teuthology.orchestra.run.smithi098.stdout:4433 write 0xea8b6a6 thru 0xea951d9 (0x9b34 bytes) 2024-09-16T13:18:06.475 INFO:teuthology.orchestra.run.smithi098.stdout:4436 write 0xcaf5f6c thru 0xcaf8790 (0x2825 bytes) 2024-09-16T13:18:06.482 INFO:teuthology.orchestra.run.smithi098.stdout:4437 read 0x10457cd thru 0x1054c9a (0xf4ce bytes) 2024-09-16T13:18:06.483 INFO:teuthology.orchestra.run.smithi098.stdout:4438 punch from 0xe973c43 to 0xe97fd27, (0xc0e4 bytes) 2024-09-16T13:18:06.483 INFO:teuthology.orchestra.run.smithi098.stdout:4439 write 0x559f136 thru 0x55a1701 (0x25cc bytes) 2024-09-16T13:18:06.486 INFO:teuthology.orchestra.run.smithi098.stdout:4440 write 0xf061c7 thru 0xf085bc (0x23f6 bytes) 2024-09-16T13:18:06.489 INFO:teuthology.orchestra.run.smithi098.stdout:4441 write 0xdbde233 thru 0xdbde410 (0x1de bytes) 2024-09-16T13:18:06.516 INFO:teuthology.orchestra.run.smithi098.stdout:4442 trunc from 0xea951da to 0x2dd5235 2024-09-16T13:18:06.536 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:06.527+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:06.545 INFO:teuthology.orchestra.run.smithi098.stdout:4443 read 0xe78eae thru 0xe7f849 (0x699c bytes) 2024-09-16T13:18:06.574 INFO:teuthology.orchestra.run.smithi098.stdout:4444 write 0xe378e1d thru 0xe37c1be (0x33a2 bytes) 2024-09-16T13:18:06.580 INFO:teuthology.orchestra.run.smithi098.stdout:4446 write 0x8ce1e91 thru 0x8cea3b1 (0x8521 bytes) 2024-09-16T13:18:06.581 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:06.581 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:06.586 INFO:teuthology.orchestra.run.smithi098.stdout:4447 read 0x1b8c7ff thru 0x1b90b03 (0x4305 bytes) 2024-09-16T13:18:06.587 INFO:teuthology.orchestra.run.smithi098.stdout:4448 read 0xc7daa82 thru 0xc7e6011 (0xb590 bytes) 2024-09-16T13:18:06.587 INFO:teuthology.orchestra.run.smithi098.stdout:4450 write 0x35d14ae thru 0x35dd447 (0xbf9a bytes) 2024-09-16T13:18:06.591 INFO:teuthology.orchestra.run.smithi098.stdout:4451 write 0x2bbfbe3 thru 0x2bcef29 (0xf347 bytes) 2024-09-16T13:18:06.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:06.594 INFO:teuthology.orchestra.run.smithi098.stdout:4452 read 0xafd083 thru 0xb095c6 (0xc544 bytes) 2024-09-16T13:18:06.611 INFO:teuthology.orchestra.run.smithi098.stdout:4453 trunc from 0xe37c1bf to 0x5a3dd13 2024-09-16T13:18:06.633 INFO:teuthology.orchestra.run.smithi098.stdout:4454 read 0x4580a7d thru 0x4583ed1 (0x3455 bytes) 2024-09-16T13:18:06.634 INFO:teuthology.orchestra.run.smithi098.stdout:4455 read 0x3663c31 thru 0x3667119 (0x34e9 bytes) 2024-09-16T13:18:06.634 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:06.624+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:06.646 INFO:teuthology.orchestra.run.smithi098.stdout:4457 write 0x5ef5e0f thru 0x5f001e6 (0xa3d8 bytes) 2024-09-16T13:18:06.656 INFO:teuthology.orchestra.run.smithi098.stdout:4458 read 0x3721a46 thru 0x372d6f6 (0xbcb1 bytes) 2024-09-16T13:18:06.657 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.657 INFO:teuthology.orchestra.run.smithi098.stdout:4461 punch from 0x8a158e to 0x8a73af, (0x5e21 bytes) 2024-09-16T13:18:06.666 INFO:teuthology.orchestra.run.smithi098.stdout:4462 trunc from 0x5f001e7 to 0xb7f483f 2024-09-16T13:18:06.678 INFO:teuthology.orchestra.run.smithi098.stdout:4463 read 0x6d0584 thru 0x6d7fa0 (0x7a1d bytes) 2024-09-16T13:18:06.696 INFO:teuthology.orchestra.run.smithi098.stdout:4464 writesame 0xdba78d5 thru 0xdba97c4 data_size 0x3(0x1ef0 bytes) 2024-09-16T13:18:06.703 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:06.703 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:06.709 INFO:teuthology.orchestra.run.smithi098.stdout:4465 trunc from 0xdba97c5 to 0x5dddfe7 2024-09-16T13:18:06.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:06.742 INFO:teuthology.orchestra.run.smithi098.stdout:4466 writesame 0x8f83b08 thru 0x8f8fe08 data_size 0x1(0xc301 bytes) 2024-09-16T13:18:06.742 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.742 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.742 INFO:teuthology.orchestra.run.smithi098.stdout:4470 read 0x731f33d thru 0x7328bf6 (0x98ba bytes) 2024-09-16T13:18:06.743 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:06.732+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:06.743 INFO:teuthology.orchestra.run.smithi098.stdout:4471 punch from 0x8e9c07e to 0x8ea4687, (0x8609 bytes) 2024-09-16T13:18:06.743 INFO:teuthology.orchestra.run.smithi098.stdout:4472 writesame 0x724b65f thru 0x7254bc8 data_size 0x96(0x956a bytes) 2024-09-16T13:18:06.750 INFO:teuthology.orchestra.run.smithi098.stdout:4473 read 0x7f83062 thru 0x7f8f02d (0xbfcc bytes) 2024-09-16T13:18:06.750 INFO:teuthology.orchestra.run.smithi098.stdout:4475 read 0x293599c thru 0x293dcba (0x831f bytes) 2024-09-16T13:18:06.751 INFO:teuthology.orchestra.run.smithi098.stdout:4476 read 0x74f74e8 thru 0x74fad4f (0x3868 bytes) 2024-09-16T13:18:06.752 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:06.752 INFO:teuthology.orchestra.run.smithi098.stdout:4478 read 0x5a7059c thru 0x5a76618 (0x607d bytes) 2024-09-16T13:18:06.753 INFO:teuthology.orchestra.run.smithi098.stdout:4479 clone 30 order 20 su 4096 sc 2 2024-09-16T13:18:06.835 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:06.828+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:06.845 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:06.845 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:06.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:06.936 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:06.927+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:06.995 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:06.995 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:07.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:07.037 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:07.028+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:07.104 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:07.104 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:07.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:07.137 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:07.128+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:07.234 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:07.234 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:07.238 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:07.228+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:07.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:07.339 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:07.328+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:07.350 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:07.350 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:07.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:07.440 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:07.431+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:07.461 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:07.461 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:07.467 INFO:teuthology.orchestra.run.smithi098.stdout:leaving image image_client.0-clone29 intact 2024-09-16T13:18:07.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:07.541 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:07.535+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:07.565 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:07.565 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:07.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:07.641 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:07.635+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:18:07.724 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:07.724 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:07.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:07.742 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:07.735+0000 7f01e473d700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 22278) UID: 0 2024-09-16T13:18:07.840 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:07.840 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:07.843 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:07.836+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:07.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:07.944 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:07.936+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:07.997 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:07.998 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:08.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:08.024 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #28, image image_client.0-clone28 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent29 2024-09-16T13:18:08.045 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:08.039+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:08.053 INFO:teuthology.orchestra.run.smithi098.stdout:4480 read 0x4517fe0 thru 0x4521c0b (0x9c2c bytes) 2024-09-16T13:18:08.058 INFO:teuthology.orchestra.run.smithi098.stdout:4481 read 0x583fc0d thru 0x58469bf (0x6db3 bytes) 2024-09-16T13:18:08.061 INFO:teuthology.orchestra.run.smithi098.stdout:4482 punch from 0x3ee76e4 to 0x3ef5316, (0xdc32 bytes) 2024-09-16T13:18:08.093 INFO:teuthology.orchestra.run.smithi098.stdout:4483 write 0xd84f394 thru 0xd8590d2 (0x9d3f bytes) 2024-09-16T13:18:08.102 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:08.102 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:08.103 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:08.103 INFO:teuthology.orchestra.run.smithi098.stdout:4485 punch from 0x30e1dfe to 0x30e290e, (0xb10 bytes) 2024-09-16T13:18:08.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:08.114 INFO:teuthology.orchestra.run.smithi098.stdout:4488 write 0xdd5064e thru 0xdd58862 (0x8215 bytes) 2024-09-16T13:18:08.122 INFO:teuthology.orchestra.run.smithi098.stdout:4489 read 0x2eb8f0e thru 0x2ec1333 (0x8426 bytes) 2024-09-16T13:18:08.140 INFO:teuthology.orchestra.run.smithi098.stdout:4490 trunc from 0xdd58863 to 0x6cf4484 2024-09-16T13:18:08.157 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:08.151+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:08.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:18:08.175 INFO:teuthology.orchestra.run.smithi098.stdout:4493 read 0x10597f7 thru 0x1059e63 (0x66d bytes) 2024-09-16T13:18:08.175 INFO:teuthology.orchestra.run.smithi098.stdout:4495 write 0x6788e79 thru 0x678f8ae (0x6a36 bytes) 2024-09-16T13:18:08.197 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:08.214 INFO:teuthology.orchestra.run.smithi098.stdout:4497 write 0xc71502f thru 0xc71c652 (0x7624 bytes) 2024-09-16T13:18:08.216 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:08.216 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:08.221 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:08.221 INFO:teuthology.orchestra.run.smithi098.stdout:4500 read 0x4f648ea thru 0x4f7307b (0xe792 bytes) 2024-09-16T13:18:08.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:18:08.238 INFO:teuthology.orchestra.run.smithi098.stdout:4501 trunc from 0xc71c653 to 0x64cf0e2 2024-09-16T13:18:08.247 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:08.239+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:08.277 INFO:teuthology.orchestra.run.smithi098.stdout:4502 punch from 0x39d7fb3 to 0x39da301, (0x234e bytes) 2024-09-16T13:18:08.277 INFO:teuthology.orchestra.run.smithi098.stdout:4504 read 0x77e4a5 thru 0x78590d (0x7469 bytes) 2024-09-16T13:18:08.278 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:08.278 INFO:teuthology.orchestra.run.smithi098.stdout:4509 read 0x5ead3b0 thru 0x5ead465 (0xb6 bytes) 2024-09-16T13:18:08.278 INFO:teuthology.orchestra.run.smithi098.stdout:4510 read 0x2bc4710 thru 0x2bc631c (0x1c0d bytes) 2024-09-16T13:18:08.279 INFO:teuthology.orchestra.run.smithi098.stdout:4511 read 0x150b57d thru 0x151417f (0x8c03 bytes) 2024-09-16T13:18:08.281 INFO:teuthology.orchestra.run.smithi098.stdout:4512 clone 31 order 23 su 2097152 sc 14 2024-09-16T13:18:08.324 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:08.324 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:08.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:18:08.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:08.377 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:08.371+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:08.429 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:08.429 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:08.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:18:08.448 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:08.440+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:08.465 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:08.485 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:08.506 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:08.534 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:08.542 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:08.542 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:08.549 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:08.540+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:08.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:08.556 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:08.577 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:08.598 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:08.625 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:08.628 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:08.628 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:08.628 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:08.628 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:08.628 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:08.628 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:08.628 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:08.628 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:18:08.650 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:08.643+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:08.653 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:08.653 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:08.653 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:08.653 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:08.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:08.751 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:08.744+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:08.803 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:08.803 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:08.803 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:08.803 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:08.803 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:08.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:08.851 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:08.844+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:08.910 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:11778 3.1 3:80993a34:::rbd_data.112d21cb8818.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:18:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.385954161000001, 2024-09-16T13:18:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.022495198000000001, 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 11778 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518950+0000", 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.518952+0000", 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.711e-06 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522261+0000", 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033087429999999998 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522296+0000", 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5335e-05 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522306+0000", 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0074999999999999e-05 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522379+0000", 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2554999999999999e-05 2024-09-16T13:18:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.522572+0000", 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019323100000000001 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.523266+0000", 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00069379000000000001 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.534086+0000", 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010820112999999999 2024-09-16T13:18:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541312+0000", 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0072264440000000003 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541394+0000", 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1710000000000002e-05 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.541446+0000", 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1490999999999999e-05 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35550 3.1 3:80899ff9:::rbd_data.112d94eb720f.0000000000000001:20 [sparse-read 4390912~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860639+0000", 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.044265373, 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017927748, 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35550 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860639+0000", 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860639+0000", 2024-09-16T13:18:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860640+0000", 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.6299999999999993e-07 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860643+0000", 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.027e-06 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860652+0000", 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9060000000000008e-06 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860657+0000", 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7380000000000002e-06 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.878436+0000", 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017779281000000001 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.878494+0000", 2024-09-16T13:18:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8156999999999999e-05 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.878567+0000", 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2675999999999993e-05 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35551 3.1 3:80899ff9:::rbd_data.112d94eb720f.0000000000000001:20 [sparse-read 5111808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860662+0000", 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0442425809999998, 2024-09-16T13:18:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018106397, 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35551 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860662+0000", 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860662+0000", 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860663+0000", 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3399999999999998e-07 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860666+0000", 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9059999999999998e-06 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860673+0000", 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8990000000000004e-06 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860677+0000", 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3259999999999997e-06 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.878606+0000", 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017929581 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.878681+0000", 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4684999999999994e-05 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.878768+0000", 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7266000000000001e-05 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35552 3.1 3:80899ff9:::rbd_data.112d94eb720f.0000000000000001:20 [sparse-read 5570560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860681+0000", 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0442233719999998, 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018303594999999999, 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35552 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860681+0000", 2024-09-16T13:18:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860681+0000", 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860682+0000", 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5999999999999999e-07 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860684+0000", 2024-09-16T13:18:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3910000000000001e-06 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860691+0000", 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5540000000000003e-06 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860695+0000", 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1370000000000004e-06 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.878796+0000", 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018101638999999999 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.878875+0000", 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8881999999999995e-05 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.878985+0000", 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010953199999999999 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.928 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35553 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860699+0000", 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0442054930000002, 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018322635, 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35553 2024-09-16T13:18:08.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860699+0000", 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860699+0000", 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860699+0000", 2024-09-16T13:18:08.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1300000000000001e-07 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860701+0000", 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9570000000000001e-06 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860707+0000", 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8930000000000002e-06 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860711+0000", 2024-09-16T13:18:08.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7239999999999998e-06 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879015+0000", 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018303460000000001 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879022+0000", 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1879999999999996e-06 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35554 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860715+0000", 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.044189517, 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018336466999999999, 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35554 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860715+0000", 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860715+0000", 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860716+0000", 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860718+0000", 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2819999999999999e-06 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860725+0000", 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7349999999999999e-06 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860728+0000", 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7390000000000001e-06 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879046+0000", 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01831725 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879052+0000", 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9939999999999997e-06 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.936 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35555 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860733+0000", 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0441718820000001, 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018346068, 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35555 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860733+0000", 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860733+0000", 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860733+0000", 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860741+0000", 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7689999999999996e-06 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860750+0000", 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8589999999999994e-06 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860755+0000", 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8779999999999997e-06 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879073+0000", 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018317797 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879079+0000", 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2600000000000002e-06 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.940 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35556 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860759+0000", 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0441451719999999, 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018349569, 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35556 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860759+0000", 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860759+0000", 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860760+0000", 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5400000000000002e-07 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860762+0000", 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1110000000000002e-06 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860768+0000", 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7590000000000003e-06 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860772+0000", 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7149999999999997e-06 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879100+0000", 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018328012000000001 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879109+0000", 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5180000000000004e-06 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35557 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860777+0000", 2024-09-16T13:18:08.944 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0441278999999999, 2024-09-16T13:18:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018364235, 2024-09-16T13:18:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35557 2024-09-16T13:18:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860777+0000", 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860777+0000", 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860777+0000", 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9800000000000004e-07 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860780+0000", 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3460000000000001e-06 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860786+0000", 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3029999999999996e-06 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860790+0000", 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7469999999999999e-06 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879134+0000", 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018344671999999999 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879141+0000", 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.669e-06 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35558 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860794+0000", 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0441104010000002, 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018377529, 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35558 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860794+0000", 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860794+0000", 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860795+0000", 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.89e-07 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860798+0000", 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8589999999999998e-06 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860805+0000", 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8029999999999999e-06 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860809+0000", 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7969999999999998e-06 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879166+0000", 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018356708999999999 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879172+0000", 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9719999999999997e-06 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35559 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860814+0000", 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0440908879999999, 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018383215000000001, 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35559 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860814+0000", 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860814+0000", 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860814+0000", 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3500000000000002e-07 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860816+0000", 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3020000000000002e-06 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860823+0000", 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6159999999999998e-06 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860827+0000", 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879191+0000", 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018364427999999999 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879197+0000", 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35560 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.860831+0000", 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0440731950000002, 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018398145000000001, 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35560 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860831+0000", 2024-09-16T13:18:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860831+0000", 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860832+0000", 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4400000000000001e-07 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860834+0000", 2024-09-16T13:18:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5579999999999999e-06 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860841+0000", 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9619999999999996e-06 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.860845+0000", 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0509999999999998e-06 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879224+0000", 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018378827 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879230+0000", 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3029999999999999e-06 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35561 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.861021+0000", 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0438832530000002, 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018233951000000002, 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35561 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861021+0000", 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861021+0000", 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861023+0000", 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.282e-06 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861027+0000", 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7049999999999998e-06 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861041+0000", 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3521e-05 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861048+0000", 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8700000000000003e-06 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879249+0000", 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018201654000000001 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879255+0000", 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9190000000000004e-06 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35562 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.861055+0000", 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0438490800000002, 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018225976000000001, 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35562 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861055+0000", 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861055+0000", 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861056+0000", 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861059+0000", 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.264e-06 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861069+0000", 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.696e-06 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861074+0000", 2024-09-16T13:18:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5329999999999996e-06 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879276+0000", 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018202614999999998 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879281+0000", 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2569999999999998e-06 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35563 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.861079+0000", 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0438259099999998, 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018236359000000001, 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35563 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861079+0000", 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861079+0000", 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861079+0000", 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0100000000000005e-07 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861082+0000", 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6340000000000001e-06 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861089+0000", 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1470000000000004e-06 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861093+0000", 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9219999999999999e-06 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879309+0000", 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018216085999999999 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879315+0000", 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0689999999999998e-06 2024-09-16T13:18:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35564 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.861098+0000", 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0438069130000001, 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018245010999999998, 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35564 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861098+0000", 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861098+0000", 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861098+0000", 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861109+0000", 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1229999999999999e-05 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861117+0000", 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3900000000000004e-06 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861121+0000", 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9450000000000003e-06 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879338+0000", 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018217016999999999 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879343+0000", 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9819999999999999e-06 2024-09-16T13:18:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35565 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.861125+0000", 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0437790979999999, 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018244182000000001, 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35565 2024-09-16T13:18:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861125+0000", 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861125+0000", 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861126+0000", 2024-09-16T13:18:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5999999999999999e-07 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861128+0000", 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4719999999999998e-06 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861135+0000", 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1759999999999998e-06 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861140+0000", 2024-09-16T13:18:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4179999999999996e-06 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879364+0000", 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018223763 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879370+0000", 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8930000000000002e-06 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35566 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.861150+0000", 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.043755005, 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018247712999999999, 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35566 2024-09-16T13:18:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861150+0000", 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861150+0000", 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861150+0000", 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2500000000000004e-07 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861153+0000", 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6730000000000001e-06 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861160+0000", 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8900000000000001e-06 2024-09-16T13:18:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861165+0000", 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7509999999999999e-06 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879391+0000", 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018226375999999999 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879397+0000", 2024-09-16T13:18:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1979999999999998e-06 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35567 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.861170+0000", 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0437349359999999, 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018258244999999999, 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35567 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861170+0000", 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861170+0000", 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861170+0000", 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861172+0000", 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861178+0000", 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2029999999999997e-06 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861182+0000", 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7689999999999998e-06 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879422+0000", 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01823982 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879428+0000", 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8590000000000001e-06 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35568 3.1 3.9ff99101 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.861186+0000", 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.0437182620000001, 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.018268550000000001, 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35568 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861186+0000", 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861186+0000", 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861187+0000", 2024-09-16T13:18:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861189+0000", 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9659999999999999e-06 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861195+0000", 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0580000000000003e-06 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.861478+0000", 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028361 2024-09-16T13:18:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879449+0000", 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.017970526000000001 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.879455+0000", 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.015e-06 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:08.993 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:08.994 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:08.994 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:08.991+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:09.095 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:09.091+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:09.139 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:09.139 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:09.139 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:09.148 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:09.174 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:18:09.196 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:09.187+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:09.237 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:18:09.237 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:18:09.237 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:18:09.238 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T13:18:09.238 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:18:09.248 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:09.248 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:09.249 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:09.249 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:09.249 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:09.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:09.297 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:09.291+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:09.312 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T13:18:09.312 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-09-16T13:18:09.312 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:18:09.312 DEBUG:teuthology.orchestra.run.smithi151:> type ceph-objectstore-tool 2024-09-16T13:18:09.318 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T13:18:09.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-16T13:18:09.356 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:09.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11223 3.8 e40/23 3:10243a59:::rbd_data.112d21cb8818.0000000000000001:head v 40'368, mlcod=39'365)", 2024-09-16T13:18:09.357 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:18:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.885601317999999, 2024-09-16T13:18:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01312405, 2024-09-16T13:18:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.358 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466702+0000", 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.466704+0000", 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.792e-06 2024-09-16T13:18:09.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471144+0000", 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044394830000000001 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471180+0000", 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5884999999999999e-05 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471194+0000", 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4258e-05 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471273+0000", 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9159999999999994e-05 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.471305+0000", 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2236000000000001e-05 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479753+0000", 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0084474049999999998 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.479826+0000", 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3831000000000005e-05 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11776 3.1a e40/23 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head v 40'72, mlcod=40'70)", 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.824704521000001, 2024-09-16T13:18:09.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.016595333, 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527599+0000", 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.527601+0000", 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.522e-06 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536775+0000", 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0091746569999999993 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536818+0000", 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2435000000000003e-05 2024-09-16T13:18:09.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536828+0000", 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0624e-05 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536931+0000", 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010280000000000001 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.536961+0000", 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0060999999999999e-05 2024-09-16T13:18:09.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544146+0000", 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0071846339999999996 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.544195+0000", 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8600000000000002e-05 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.366 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13719 3.19 3:99fbe691:::rbd_data.112df1cc59bd.000000000000000c:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.07153495, 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011713708999999999, 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13719 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280769+0000", 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280771+0000", 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9240000000000001e-06 2024-09-16T13:18:09.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280782+0000", 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1642e-05 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280806+0000", 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4119000000000001e-05 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280815+0000", 2024-09-16T13:18:09.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7150000000000004e-06 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012396900000000001 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281142+0000", 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000203298 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.281540+0000", 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00039763500000000002 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.286598+0000", 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0050579090000000002 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292380+0000", 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057823390000000001 2024-09-16T13:18:09.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292433+0000", 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2612999999999999e-05 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292482+0000", 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9546000000000003e-05 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13720 3.19 3:99acede8:::rbd_data.112df1cc59bd.000000000000000d:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.071471758000001, 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011830879000000001, 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13720 2024-09-16T13:18:09.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280832+0000", 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4799999999999998e-07 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280836+0000", 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.703e-06 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280844+0000", 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280849+0000", 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.254e-06 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282739+0000", 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.001889672 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.282875+0000", 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013641 2024-09-16T13:18:09.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.283217+0000", 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00034147099999999998 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.288528+0000", 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053110140000000002 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:09.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292558+0000", 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040304429999999999 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292620+0000", 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1588999999999996e-05 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.292663+0000", 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2922e-05 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.378 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:13726 3.19 3:9e45447e:::rbd_data.112df1cc59bd.0000000000000034:head [delete] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.071371044999999, 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012171434, 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 13726 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280933+0000", 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280934+0000", 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9299999999999996e-07 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280939+0000", 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280950+0000", 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1141000000000001e-05 2024-09-16T13:18:09.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.280955+0000", 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5560000000000001e-06 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284230+0000", 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032752789999999999 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284286+0000", 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6456999999999998e-05 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.284430+0000", 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014382899999999999 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.291179+0000", 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0067490099999999997 2024-09-16T13:18:09.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293014+0000", 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018347719999999999 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293069+0000", 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5325999999999998e-05 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:23.293104+0000", 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4892999999999999e-05 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14663 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2228224~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716844692999999, 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098295339999999991, 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.385 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14663 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635459+0000", 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635460+0000", 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1500000000000004e-07 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635463+0000", 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.478e-06 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635472+0000", 2024-09-16T13:18:09.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4230000000000005e-06 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635498+0000", 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6543e-05 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645115+0000", 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096171710000000007 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645193+0000", 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7245999999999999e-05 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645289+0000", 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5958000000000005e-05 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.389 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14664 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716798203000003, 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010033297, 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14664 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635505+0000", 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635506+0000", 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5099999999999998e-07 2024-09-16T13:18:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635511+0000", 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7820000000000001e-06 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635520+0000", 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5969999999999992e-06 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635529+0000", 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3249999999999997e-06 2024-09-16T13:18:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645337+0000", 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0098082589999999997 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645427+0000", 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9922999999999999e-05 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645539+0000", 2024-09-16T13:18:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011156 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14665 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716768975000001, 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010434005999999999, 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.394 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14665 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635535+0000", 2024-09-16T13:18:09.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635538+0000", 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8329999999999998e-06 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635546+0000", 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0760000000000003e-06 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635551+0000", 2024-09-16T13:18:09.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6160000000000004e-06 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645677+0000", 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010126546 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645821+0000", 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014406799999999999 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.645969+0000", 2024-09-16T13:18:09.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014730900000000001 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14678 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 0~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716509531, 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012023611999999999, 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.398 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14678 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635794+0000", 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635797+0000", 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7039999999999999e-06 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635810+0000", 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3343000000000001e-05 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635845+0000", 2024-09-16T13:18:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4751e-05 2024-09-16T13:18:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635863+0000", 2024-09-16T13:18:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8210999999999999e-05 2024-09-16T13:18:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646005+0000", 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010141766999999999 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.646070+0000", 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5324e-05 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647818+0000", 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017475119999999999 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14679 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1835008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:18:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716421824999998, 2024-09-16T13:18:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012069985, 2024-09-16T13:18:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14679 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635882+0000", 2024-09-16T13:18:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635883+0000", 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3239999999999999e-06 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635889+0000", 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635905+0000", 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6022999999999999e-05 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635911+0000", 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3740000000000003e-06 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647849+0000", 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.011937906999999999 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647895+0000", 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5507999999999999e-05 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647952+0000", 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7132000000000001e-05 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14680 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2359296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:18:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716385543000001, 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012172090999999999, 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14680 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635918+0000", 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635919+0000", 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8299999999999995e-07 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635922+0000", 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.027e-06 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635931+0000", 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6230000000000003e-06 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635936+0000", 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0069999999999997e-06 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.647981+0000", 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012045273 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648033+0000", 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2024000000000002e-05 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648090+0000", 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7154000000000002e-05 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14681 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4718592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716362545999999, 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012312528, 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14681 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635941+0000", 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635942+0000", 2024-09-16T13:18:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7599999999999996e-07 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635959+0000", 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6954999999999999e-05 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635968+0000", 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8929999999999994e-06 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635973+0000", 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0170000000000004e-06 2024-09-16T13:18:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648120+0000", 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012146847000000001 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648170+0000", 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0368000000000003e-05 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648254+0000", 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.3671999999999994e-05 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14694 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 131072~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716325472000001, 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012792843999999999, 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14694 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635978+0000", 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635979+0000", 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0699999999999996e-07 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635982+0000", 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8200000000000001e-06 2024-09-16T13:18:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635990+0000", 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0239999999999997e-06 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.635995+0000", 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8999999999999997e-06 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648296+0000", 2024-09-16T13:18:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0123009 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648397+0000", 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101042 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648771+0000", 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00037435099999999998 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14695 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 1966080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716303453999998, 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012992136, 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14695 2024-09-16T13:18:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636000+0000", 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636001+0000", 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0999999999999998e-07 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636004+0000", 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5959999999999998e-06 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636012+0000", 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636016+0000", 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9269999999999996e-06 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648804+0000", 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012787543 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648873+0000", 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9263000000000004e-05 2024-09-16T13:18:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.648992+0000", 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119089 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14696 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 2490368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716281762999998, 2024-09-16T13:18:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013132612, 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14696 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636022+0000", 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636023+0000", 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7300000000000005e-07 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636025+0000", 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.407e-06 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636033+0000", 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0129999999999993e-06 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636042+0000", 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1540000000000008e-06 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649027+0000", 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.012985096 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649089+0000", 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1641000000000002e-05 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649155+0000", 2024-09-16T13:18:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5528e-05 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:14697 3.19 3:99202132:::rbd_data.112d9055945.0000000000000001:e [sparse-read 4849664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.716255861999997, 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013321177999999999, 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 14697 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636048+0000", 2024-09-16T13:18:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636049+0000", 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636051+0000", 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.57e-06 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636059+0000", 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.9440000000000005e-06 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.636067+0000", 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0460000000000006e-06 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649182+0000", 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.01311522 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649245+0000", 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2930999999999998e-05 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:26.649369+0000", 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123681 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:29335 3.0 3:063d9f84:::rbd_data.112dab4f0555.0000000000000034:1b [read 1671168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:56.285651+0000", 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.066652384999999, 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0097858489999999992, 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 29335 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285651+0000", 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285651+0000", 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285652+0000", 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285654+0000", 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3329999999999999e-06 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285660+0000", 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2480000000000001e-06 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285664+0000", 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3280000000000002e-06 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295336+0000", 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0096724859999999992 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295390+0000", 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3819000000000001e-05 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295437+0000", 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7216999999999999e-05 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:29336 3.0 3:063d9f84:::rbd_data.112dab4f0555.0000000000000034:1b [read 1867776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:56.285667+0000", 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.066636243, 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0099780400000000005, 2024-09-16T13:18:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 29336 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285667+0000", 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285667+0000", 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285668+0000", 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2500000000000001e-07 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285688+0000", 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9958000000000001e-05 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285699+0000", 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1233000000000001e-05 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285704+0000", 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7770000000000002e-06 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295465+0000", 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0097615080000000003 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295536+0000", 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0931000000000002e-05 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295645+0000", 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109208 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:29337 3.0 3:063d9f84:::rbd_data.112dab4f0555.0000000000000034:1b [read 2064384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:56.285709+0000", 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.066594712000001, 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010216506, 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 29337 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285709+0000", 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285709+0000", 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285710+0000", 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:18:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285712+0000", 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.39e-06 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285721+0000", 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4630000000000007e-06 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.285726+0000", 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:18:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295757+0000", 2024-09-16T13:18:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010031093 2024-09-16T13:18:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295852+0000", 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5538000000000006e-05 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:56.295925+0000", 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3323000000000003e-05 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:34450 3.19 3:9ae947c7:::rbd_object_map.112d94eb720f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.410040+0000", 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.9422639820000001, 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.010154194, 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 34450 2024-09-16T13:18:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.410040+0000", 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.410040+0000", 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.410040+0000", 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.410043+0000", 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7580000000000001e-06 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.410056+0000", 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3176000000000001e-05 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.410060+0000", 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7459999999999998e-06 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.414652+0000", 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045922879999999999 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.414750+0000", 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8202000000000006e-05 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.414979+0000", 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022863199999999999 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.419785+0000", 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004805559 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.420072+0000", 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00028689600000000003 2024-09-16T13:18:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.420145+0000", 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3744000000000004e-05 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.420194+0000", 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8622999999999998e-05 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.454 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:09.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:09.456 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:09.447+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:09.557 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:09.551+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:09.560 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:09.560 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:09.560 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:09.560 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:09.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:09.658 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:09.651+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:09.671 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:09.671 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:09.671 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:09.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:09.759 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:09.751+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:09.781 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:09.781 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:09.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:09.781 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:09.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.781 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:18:09.781 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:09.781 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.768430444000003, 2024-09-16T13:18:09.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:18:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.785 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.767644945000001, 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:09.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:18:09.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.766478023000005, 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:09.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:18:09.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.763603391999993, 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:18:09.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:18:09.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.760589651999993, 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:18:09.802 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.735385449000006, 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:18:09.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:18:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:18:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:18:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:18:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 64.969236559999999, 2024-09-16T13:18:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:18:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:18:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:18:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5115 3.4 e34/23 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head v 34'14, mlcod=31'12)", 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.984188453999998, 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029400680000000001, 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:18:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:18:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791635+0000", 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4200000000000005e-07 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791643+0000", 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.157e-06 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791660+0000", 2024-09-16T13:18:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6246e-05 2024-09-16T13:18:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791666+0000", 2024-09-16T13:18:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-16T13:18:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791743+0000", 2024-09-16T13:18:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7000000000000001e-05 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791756+0000", 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2729e-05 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794523+0000", 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027675949999999999 2024-09-16T13:18:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794575+0000", 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1082000000000003e-05 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 57.717134256999998, 2024-09-16T13:18:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:18:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:18:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:18:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 57.716524817, 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:18:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:18:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:18:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:18:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:18:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:18:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:18:09.830 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 57.219241003, 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:09.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:18:09.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:18:09.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:18:09.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:18:09.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.835 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 57.218140187000003, 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:18:09.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:18:09.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:18:09.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:18:09.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.841 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:18:09.841 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:09.841 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 57.218000547000003, 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:18:09.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:18:09.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 57.217500016999999, 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:09.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:18:09.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:18:09.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:18:09.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.222330630999998, 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:18:09.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:18:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.857 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.965916790000001, 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:18:09.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:18:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:18:09.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:18:09.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:18:09.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:18:09.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.862 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.353353683000002, 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:18:09.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:18:09.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.867 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:22854 3.9 e57/23 3:924a2914:::rbd_data.112db5030c67.0000000000000008:head v 57'198, mlcod=57'196)", 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.234833119000001, 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029464980000000001, 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540995+0000", 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:18:09.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541018+0000", 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2382999999999999e-05 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541079+0000", 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1773999999999995e-05 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541094+0000", 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4414e-05 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.542159+0000", 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010653749999999999 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.542208+0000", 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8517000000000003e-05 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.543797+0000", 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015893210000000001 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.543936+0000", 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000139367 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.515247513, 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:18:09.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.151120179999999, 2024-09-16T13:18:09.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:09.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:09.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:18:09.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:09.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:09.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:18:09.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:18:09.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.882 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:09.882 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:09.882 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:09.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:09.884 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:09.880+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:09.984 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:09.975+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:10.034 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:10.034 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:10.034 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:10.034 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:10.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:10.085 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:10.080+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:10.144 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:10.144 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:10.145 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:10.145 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:10.145 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:10.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:10.203 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:10.200+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:10.253 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:10.253 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:10.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:10.253 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:10.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.253 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35728 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.253 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863526+0000", 2024-09-16T13:18:10.253 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3844785220000002, 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020675761000000001, 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35728 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863526+0000", 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863526+0000", 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863526+0000", 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863528+0000", 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8929999999999999e-06 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863533+0000", 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3360000000000002e-06 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863537+0000", 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3890000000000001e-06 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884196+0000", 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020658916999999999 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884202+0000", 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8590000000000001e-06 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35729 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863546+0000", 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3844579640000001, 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020686517000000001, 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35729 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863546+0000", 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863546+0000", 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863547+0000", 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9400000000000001e-07 2024-09-16T13:18:10.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863549+0000", 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0430000000000002e-06 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863555+0000", 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.755e-06 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863558+0000", 2024-09-16T13:18:10.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5580000000000001e-06 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884227+0000", 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020668995999999999 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884233+0000", 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7710000000000003e-06 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35730 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863562+0000", 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3844424719999999, 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020699299000000001, 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35730 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863562+0000", 2024-09-16T13:18:10.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863562+0000", 2024-09-16T13:18:10.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863562+0000", 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863564+0000", 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.99e-06 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863570+0000", 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5609999999999998e-06 2024-09-16T13:18:10.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863573+0000", 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.287e-06 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884255+0000", 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020682025999999999 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884261+0000", 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:18:10.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35731 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863577+0000", 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3844275479999997, 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020716910000000002, 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35731 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863577+0000", 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863577+0000", 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863577+0000", 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863579+0000", 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.984e-06 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863584+0000", 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863588+0000", 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2210000000000002e-06 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884285+0000", 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020697345999999998 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884294+0000", 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7739999999999993e-06 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35732 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863591+0000", 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3844131300000004, 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020741091999999999, 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35732 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863591+0000", 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863591+0000", 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863592+0000", 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1199999999999998e-07 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863594+0000", 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9819999999999998e-06 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863599+0000", 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3360000000000002e-06 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863603+0000", 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:18:10.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884325+0000", 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020722905999999999 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884332+0000", 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9079999999999998e-06 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35733 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863606+0000", 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3843981640000003, 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020754568000000001, 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35733 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863606+0000", 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863606+0000", 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863607+0000", 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863614+0000", 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8310000000000001e-06 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863620+0000", 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.823e-06 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863624+0000", 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884355+0000", 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020730769999999999 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884361+0000", 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9540000000000001e-06 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35734 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863628+0000", 2024-09-16T13:18:10.277 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3843765340000003, 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020762262, 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35734 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863628+0000", 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863628+0000", 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863628+0000", 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863635+0000", 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579999999999997e-06 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863641+0000", 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8139999999999997e-06 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863644+0000", 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884383+0000", 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020738989999999999 2024-09-16T13:18:10.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884390+0000", 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35735 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863648+0000", 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3843565719999997, 2024-09-16T13:18:10.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020780980000000001, 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35735 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863648+0000", 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863648+0000", 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863648+0000", 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1199999999999998e-07 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863650+0000", 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8479999999999999e-06 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863655+0000", 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863659+0000", 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.422e-06 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884422+0000", 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020762930999999998 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884429+0000", 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0199999999999997e-06 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35736 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863663+0000", 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3843418339999998, 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020794782000000001, 2024-09-16T13:18:10.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35736 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863663+0000", 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863663+0000", 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863663+0000", 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7599999999999998e-07 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863665+0000", 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9230000000000001e-06 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863670+0000", 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4129999999999997e-06 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863674+0000", 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884451+0000", 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020777711000000001 2024-09-16T13:18:10.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884457+0000", 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.02e-06 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35737 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863677+0000", 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3843270519999997, 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020816028, 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.289 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35737 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863677+0000", 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863677+0000", 2024-09-16T13:18:10.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863678+0000", 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863680+0000", 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9609999999999999e-06 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863685+0000", 2024-09-16T13:18:10.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4469999999999997e-06 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863694+0000", 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7509999999999997e-06 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884487+0000", 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020793345000000001 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884493+0000", 2024-09-16T13:18:10.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.127e-06 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35738 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863698+0000", 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3843067209999997, 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020825199999999999, 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35738 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863698+0000", 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863698+0000", 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863698+0000", 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.03e-07 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863700+0000", 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863705+0000", 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3310000000000003e-06 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863709+0000", 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.552e-06 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884516+0000", 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020807651999999999 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884523+0000", 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3709999999999996e-06 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35739 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863713+0000", 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3842918590000002, 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020845658999999999, 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35739 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863713+0000", 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863713+0000", 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863713+0000", 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:18:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863715+0000", 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9180000000000001e-06 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863721+0000", 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863724+0000", 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:18:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884546+0000", 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020821507 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884558+0000", 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2660000000000001e-05 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35740 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863728+0000", 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3842768459999997, 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020859386000000001, 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35740 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863728+0000", 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863728+0000", 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863728+0000", 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3500000000000002e-07 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863730+0000", 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863735+0000", 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2480000000000004e-06 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863738+0000", 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.585e-06 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884581+0000", 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020842362 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884587+0000", 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0669999999999997e-06 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35741 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863743+0000", 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.384261811, 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020873149000000001, 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35741 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863743+0000", 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863743+0000", 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863743+0000", 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863750+0000", 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863755+0000", 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863758+0000", 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3060000000000002e-06 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884610+0000", 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020851545999999999 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884616+0000", 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0360000000000003e-06 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35742 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863762+0000", 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3842419719999999, 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020882847999999999, 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35742 2024-09-16T13:18:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863762+0000", 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863762+0000", 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863763+0000", 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863765+0000", 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9719999999999999e-06 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863771+0000", 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863774+0000", 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2449999999999998e-06 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884638+0000", 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020863711 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884645+0000", 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2370000000000003e-06 2024-09-16T13:18:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35743 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863778+0000", 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3842262449999998, 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020895717000000001, 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35743 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863778+0000", 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863778+0000", 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863779+0000", 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9400000000000001e-07 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863781+0000", 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1090000000000001e-06 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863786+0000", 2024-09-16T13:18:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8119999999999996e-06 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863790+0000", 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8060000000000001e-06 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884668+0000", 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020877913000000001 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884674+0000", 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6829999999999996e-06 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35744 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863827+0000", 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.384177008, 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020880882, 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35744 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863827+0000", 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863827+0000", 2024-09-16T13:18:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863828+0000", 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8400000000000003e-07 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863831+0000", 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8109999999999999e-06 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863840+0000", 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9530000000000005e-06 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863844+0000", 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2259999999999999e-06 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884696+0000", 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020851919999999999 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884708+0000", 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2187999999999999e-05 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35745 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863848+0000", 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3841562280000002, 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020889345, 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35745 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863848+0000", 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863848+0000", 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863849+0000", 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863851+0000", 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.035e-06 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863856+0000", 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7459999999999997e-06 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863860+0000", 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6370000000000001e-06 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884731+0000", 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020871137000000001 2024-09-16T13:18:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884738+0000", 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2469999999999997e-06 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35746 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863864+0000", 2024-09-16T13:18:10.325 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3841404349999999, 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020908428, 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35746 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863864+0000", 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863864+0000", 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863864+0000", 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:18:10.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863871+0000", 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.302e-06 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863877+0000", 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9460000000000003e-06 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863880+0000", 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.659e-06 2024-09-16T13:18:10.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884762+0000", 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020881813999999999 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884772+0000", 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0349999999999999e-05 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.331 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35747 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863884+0000", 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.3841199849999999, 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020917018999999999, 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35747 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863884+0000", 2024-09-16T13:18:10.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863884+0000", 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863885+0000", 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863887+0000", 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9230000000000001e-06 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863892+0000", 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5670000000000002e-06 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863896+0000", 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884795+0000", 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020899636999999999 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884801+0000", 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1070000000000001e-06 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:10.335 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:10.336 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:18:10.263+0000 7f8b1ccd7300 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-16T13:18:10.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:10.338 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:10.332+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:10.438 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:10.432+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:10.449 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:10.449 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:10.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.449 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:37781 3.e 3.36f78d0e (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e82)", 2024-09-16T13:18:10.449 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:10.179877+0000", 2024-09-16T13:18:10.449 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.26706467299999997, 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.267107019, 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for peered", 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 37781 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:10.179877+0000", 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:10.179877+0000", 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:10.179879+0000", 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4880000000000001e-06 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:10.179891+0000", 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1579999999999999e-05 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:10.179912+0000", 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0616e-05 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:10.179924+0000", 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1827999999999999e-05 2024-09-16T13:18:10.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:10.179997+0000", 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3566999999999995e-05 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:10.180005+0000", 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4850000000000003e-06 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.454 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-16T13:18:10.454 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 1 2024-09-16T13:18:10.454 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:10.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:10.539 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:10.536+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:10.550 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:10.550 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:10.550 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:10.550 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:10.550 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:10.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:10.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:10.640 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:10.636+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:10.655 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:8373 3.1b 3:db7047f1:::rbd_object_map.112de4fe95f7:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.593504148999997, 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022897421000000001, 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056422+0000", 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3200000000000005e-07 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056427+0000", 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8520000000000003e-06 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056433+0000", 2024-09-16T13:18:10.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.056437+0000", 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8909999999999997e-06 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076470+0000", 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020032239 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076524+0000", 2024-09-16T13:18:10.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4475000000000003e-05 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7", 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.076996+0000", 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000471905 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.078749+0000", 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001752539 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:10.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079217+0000", 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046819400000000001 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079276+0000", 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9407000000000002e-05 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:14.079319+0000", 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3096999999999998e-05 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:11776 3.1a 3:592420bb:::rbd_data.112d21cb8818.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 53.132512773000002, 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027432719000000001, 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11776 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517413+0000", 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.517415+0000", 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5239999999999999e-06 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526328+0000", 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089138059999999998 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526362+0000", 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3775000000000001e-05 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526379+0000", 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6835000000000001e-05 2024-09-16T13:18:10.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526467+0000", 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7703999999999999e-05 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.526653+0000", 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186379 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1", 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.527624+0000", 2024-09-16T13:18:10.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097080400000000002 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.543313+0000", 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015689456000000001 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544696+0000", 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013822979999999999 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544784+0000", 2024-09-16T13:18:10.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8419999999999997e-05 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:17.544846+0000", 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1717999999999994e-05 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:13740 3.1 3:856e8e3c:::rbd_object_map.112df1cc59bd:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 47.351823979000002, 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023881870999999999, 2024-09-16T13:18:10.667 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13740 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298102+0000", 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298103+0000", 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298106+0000", 2024-09-16T13:18:10.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1559999999999999e-06 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298120+0000", 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4474999999999999e-05 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.298124+0000", 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1699999999999999e-06 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319635+0000", 2024-09-16T13:18:10.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021510129999999999 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.319762+0000", 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012746400000000001 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.320225+0000", 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046264800000000002 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321629+0000", 2024-09-16T13:18:10.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00140383 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321682+0000", 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3720000000000001e-05 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321757+0000", 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:23.321984+0000", 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022684199999999999 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35783 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 5636096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864293+0000", 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7856328179999998, 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022788797999999999, 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.673 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35783 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864293+0000", 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864293+0000", 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864293+0000", 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864296+0000", 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0310000000000001e-06 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864300+0000", 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5229999999999997e-06 2024-09-16T13:18:10.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864304+0000", 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1659999999999996e-06 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886894+0000", 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022589626000000002 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886982+0000", 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8577000000000003e-05 2024-09-16T13:18:10.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887082+0000", 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9494999999999995e-05 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35784 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 6356992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864308+0000", 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7856183290000001, 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022989999000000001, 2024-09-16T13:18:10.677 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35784 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864308+0000", 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864308+0000", 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864308+0000", 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864310+0000", 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.711e-06 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864314+0000", 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5900000000000001e-06 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864317+0000", 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1209999999999998e-06 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887128+0000", 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022810438999999998 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887190+0000", 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1963000000000003e-05 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887298+0000", 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107787 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.681 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35785 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 6815744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864322+0000", 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7856035139999999, 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023204272000000001, 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35785 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864322+0000", 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864322+0000", 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864323+0000", 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8099999999999998e-07 2024-09-16T13:18:10.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864324+0000", 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6959999999999999e-06 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864331+0000", 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4119999999999998e-06 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864336+0000", 2024-09-16T13:18:10.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.575e-06 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887322+0000", 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022985740000000001 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887418+0000", 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6275999999999998e-05 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887527+0000", 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108192 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35786 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 7536640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864340+0000", 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7855861260000001, 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0234305, 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.686 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35786 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864340+0000", 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864340+0000", 2024-09-16T13:18:10.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864340+0000", 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1199999999999998e-07 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864342+0000", 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.03e-06 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864347+0000", 2024-09-16T13:18:10.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.386e-06 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864350+0000", 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0479999999999999e-06 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887570+0000", 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023219884 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887643+0000", 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3763000000000005e-05 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887770+0000", 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012697699999999999 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35787 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 7995392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864353+0000", 2024-09-16T13:18:10.690 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7855731590000001, 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023638618, 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35787 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864353+0000", 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864353+0000", 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864353+0000", 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6199999999999999e-07 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864355+0000", 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7349999999999999e-06 2024-09-16T13:18:10.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864359+0000", 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.578e-06 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864362+0000", 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0120000000000001e-06 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887803+0000", 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023441001 2024-09-16T13:18:10.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887892+0000", 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8331999999999994e-05 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887991+0000", 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9598000000000004e-05 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35788 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 8716288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864365+0000", 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7855604359999999, 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023901288999999999, 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35788 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864365+0000", 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864365+0000", 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864366+0000", 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7399999999999999e-07 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864367+0000", 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5859999999999999e-06 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864390+0000", 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2322000000000001e-05 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864400+0000", 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0468999999999999e-05 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888035+0000", 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023634645999999999 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888140+0000", 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105233 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888267+0000", 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012665899999999999 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.698 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35789 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864452+0000", 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.785474035, 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024043689, 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35789 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.699 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864452+0000", 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864452+0000", 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864453+0000", 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.159e-06 2024-09-16T13:18:10.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864457+0000", 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.985e-06 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864467+0000", 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5880000000000004e-06 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864473+0000", 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3169999999999998e-06 2024-09-16T13:18:10.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888309+0000", 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023835582000000001 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888398+0000", 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9650999999999999e-05 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888496+0000", 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7406999999999998e-05 2024-09-16T13:18:10.702 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35790 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 9895936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864480+0000", 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7854462719999997, 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024266156000000001, 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.703 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35790 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864480+0000", 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864480+0000", 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864480+0000", 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1900000000000001e-07 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864483+0000", 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0110000000000001e-06 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864511+0000", 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7838e-05 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864523+0000", 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1405e-05 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888543+0000", 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024020047999999999 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888643+0000", 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100549 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888746+0000", 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010248599999999999 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35791 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 10354688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864545+0000", 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7853806599999995, 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024429507, 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.707 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35791 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864545+0000", 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864545+0000", 2024-09-16T13:18:10.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864546+0000", 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8100000000000003e-07 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864549+0000", 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0539999999999999e-06 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864571+0000", 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1673000000000001e-05 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864583+0000", 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2435999999999999e-05 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888776+0000", 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024192821999999999 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888867+0000", 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0777000000000002e-05 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.888975+0000", 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108164 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35792 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 11075584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864588+0000", 2024-09-16T13:18:10.711 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7853374610000001, 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024577912, 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35792 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864588+0000", 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864588+0000", 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864589+0000", 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864598+0000", 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0340000000000002e-06 2024-09-16T13:18:10.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864603+0000", 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8189999999999999e-06 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864606+0000", 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3349999999999998e-06 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889021+0000", 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024415088000000001 2024-09-16T13:18:10.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889084+0000", 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3015999999999994e-05 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889166+0000", 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2232999999999997e-05 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35793 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 11534336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864611+0000", 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7853154289999997, 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024776010000000001, 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35793 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864611+0000", 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864611+0000", 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864611+0000", 2024-09-16T13:18:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0999999999999999e-07 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864613+0000", 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9269999999999999e-06 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864620+0000", 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7490000000000001e-06 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864624+0000", 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5029999999999999e-06 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889198+0000", 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024574028000000001 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889287+0000", 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9333999999999995e-05 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889387+0000", 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9059000000000001e-05 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35794 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 12255232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864628+0000", 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7852977130000003, 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024971317999999999, 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35794 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864628+0000", 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864628+0000", 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864629+0000", 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864630+0000", 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6920000000000001e-06 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864635+0000", 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4320000000000001e-06 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864638+0000", 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.174e-06 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889437+0000", 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024798929000000001 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889495+0000", 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8111e-05 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889600+0000", 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104601 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35795 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 12713984~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864641+0000", 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7852850729999998, 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025164909999999999, 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35795 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864641+0000", 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864641+0000", 2024-09-16T13:18:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864641+0000", 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8599999999999999e-07 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864643+0000", 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8449999999999999e-06 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864663+0000", 2024-09-16T13:18:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9967e-05 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864668+0000", 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4730000000000002e-06 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889623+0000", 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024955182999999999 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889708+0000", 2024-09-16T13:18:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4919000000000005e-05 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889806+0000", 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8136999999999999e-05 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35796 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 13434880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864673+0000", 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7852528549999995, 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025315026000000001, 2024-09-16T13:18:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35796 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864673+0000", 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864673+0000", 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864674+0000", 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.32e-07 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864676+0000", 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0640000000000001e-06 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864686+0000", 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0788000000000001e-05 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864691+0000", 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.011e-06 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889832+0000", 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025140498000000001 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889886+0000", 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4552000000000002e-05 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.889988+0000", 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101681 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35797 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 13893632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864695+0000", 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.78523122, 2024-09-16T13:18:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025517855999999998, 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35797 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864695+0000", 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864695+0000", 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864695+0000", 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8099999999999998e-07 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864697+0000", 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7570000000000001e-06 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864702+0000", 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6709999999999998e-06 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864716+0000", 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4630999999999999e-05 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.890020+0000", 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025304283 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.890109+0000", 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8856000000000004e-05 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.890213+0000", 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010327699999999999 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35798 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 14614528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864720+0000", 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7852060359999999, 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025766686, 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35798 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864720+0000", 2024-09-16T13:18:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864720+0000", 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864720+0000", 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.32e-07 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864723+0000", 2024-09-16T13:18:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3910000000000001e-06 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864732+0000", 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8170000000000005e-06 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864738+0000", 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1970000000000001e-06 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.890264+0000", 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025526224 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.890373+0000", 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010896299999999999 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.890487+0000", 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113662 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35799 3.1b 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.864743+0000", 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7851826830000004, 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025957893999999999, 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35799 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864743+0000", 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864743+0000", 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864744+0000", 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6899999999999997e-07 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864747+0000", 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8779999999999998e-06 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864756+0000", 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9150000000000002e-06 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.864761+0000", 2024-09-16T13:18:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5650000000000001e-06 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.890511+0000", 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025749625000000002 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.890601+0000", 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9722999999999995e-05 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.890701+0000", 2024-09-16T13:18:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010061899999999999 2024-09-16T13:18:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:10.745 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:10.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:10.746 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:10.739+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:10.790 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:10.810 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:10.832 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:10.845 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:10.845 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:10.845 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:10.845 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:10.847 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:10.839+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:10.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:10.859 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:10.880 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:10.900 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:10.920 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:10.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:10.948 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:10.939+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:10.950 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:10.950 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:10.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:11.050 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:11.044+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:11.058 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:11.058 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:11.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:11.151 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:11.144+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:11.159 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:11.159 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:11.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:11.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:18:11.251 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:11.243+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:11.265 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:11.266 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:11.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:11.353 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:11.344+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:11.383 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:11.383 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:11.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:11.395 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone30 from 0x64cf0e2 (overlap 0x64cf0e2) to 0x0 2024-09-16T13:18:11.454 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:11.444+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:11.548 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:11.548 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:11.555 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:11.547+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:11.562 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:11.656 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:11.647+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:11.757 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:11.748+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:11.792 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:11.792 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:11.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:11.858 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:11.851+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:11.961 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:11.955+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:11.962 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:11.962 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:11.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:12.062 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:12.055+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:12.145 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:12.145 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:12.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:12.168 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:12.159+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:12.202 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:18:12.266 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:12.256+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:12.296 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:12.297 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:12.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:12.369 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:12.360+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:12.419 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:12.419 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:12.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:12.475 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:12.468+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:12.570 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:12.560+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:12.575 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:12.575 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:12.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:12.672 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:12.668+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:12.676 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:12.676 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:12.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:12.773 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:12.768+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:12.823 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:12.823 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:12.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:12.840 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #29, image image_client.0-clone29 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent30 2024-09-16T13:18:12.873 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:12.867+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:12.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:12.934 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:12.934 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:12.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:12.987 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:12.984+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:13.003 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-16T13:18:12.995+0000 7f8b1ccd7300 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-16T13:18:13.066 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:13.076 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:13.067+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:13.081 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:13.081 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:13.084 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:13.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:13.103 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:13.115 INFO:teuthology.orchestra.run.smithi098.stdout:4513 writesame 0x1c24174 thru 0x1c32824 data_size 0x1(0xe6b1 bytes) 2024-09-16T13:18:13.129 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:13.176 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:13.172+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:13.187 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:13.189 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:13.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:13.189 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:13.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.189 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35728 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.189 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863526+0000", 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3221571479999996, 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020675761000000001, 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35728 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863526+0000", 2024-09-16T13:18:13.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863526+0000", 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863526+0000", 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863528+0000", 2024-09-16T13:18:13.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8929999999999999e-06 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863533+0000", 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3360000000000002e-06 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863537+0000", 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3890000000000001e-06 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884196+0000", 2024-09-16T13:18:13.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020658916999999999 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884202+0000", 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8590000000000001e-06 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35729 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863546+0000", 2024-09-16T13:18:13.193 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3221365900000004, 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020686517000000001, 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35729 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863546+0000", 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863546+0000", 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863547+0000", 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9400000000000001e-07 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863549+0000", 2024-09-16T13:18:13.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0430000000000002e-06 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863555+0000", 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.755e-06 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863558+0000", 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5580000000000001e-06 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884227+0000", 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020668995999999999 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884233+0000", 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7710000000000003e-06 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35730 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863562+0000", 2024-09-16T13:18:13.197 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3221210980000002, 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020699299000000001, 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35730 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863562+0000", 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863562+0000", 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863562+0000", 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2899999999999999e-07 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863564+0000", 2024-09-16T13:18:13.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.99e-06 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863570+0000", 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5609999999999998e-06 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863573+0000", 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.287e-06 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884255+0000", 2024-09-16T13:18:13.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020682025999999999 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884261+0000", 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35731 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863577+0000", 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.322106174, 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020716910000000002, 2024-09-16T13:18:13.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35731 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863577+0000", 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863577+0000", 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863577+0000", 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863579+0000", 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.984e-06 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863584+0000", 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.242e-06 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863588+0000", 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2210000000000002e-06 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884285+0000", 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020697345999999998 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884294+0000", 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7739999999999993e-06 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35732 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863591+0000", 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3220917559999998, 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020741091999999999, 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.205 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35732 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863591+0000", 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863591+0000", 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863592+0000", 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1199999999999998e-07 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863594+0000", 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9819999999999998e-06 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863599+0000", 2024-09-16T13:18:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3360000000000002e-06 2024-09-16T13:18:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863603+0000", 2024-09-16T13:18:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:18:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884325+0000", 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020722905999999999 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884332+0000", 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9079999999999998e-06 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35733 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.209 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863606+0000", 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3220767899999997, 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020754568000000001, 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35733 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863606+0000", 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863606+0000", 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863607+0000", 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863614+0000", 2024-09-16T13:18:13.211 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8310000000000001e-06 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863620+0000", 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.823e-06 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863624+0000", 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8149999999999999e-06 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884355+0000", 2024-09-16T13:18:13.212 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020730769999999999 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884361+0000", 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9540000000000001e-06 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35734 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863628+0000", 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3220551599999997, 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020762262, 2024-09-16T13:18:13.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35734 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863628+0000", 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.214 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863628+0000", 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863628+0000", 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863635+0000", 2024-09-16T13:18:13.215 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579999999999997e-06 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863641+0000", 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8139999999999997e-06 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863644+0000", 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884383+0000", 2024-09-16T13:18:13.216 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020738989999999999 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884390+0000", 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35735 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863648+0000", 2024-09-16T13:18:13.217 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.322035198, 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020780980000000001, 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35735 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863648+0000", 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863648+0000", 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863648+0000", 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1199999999999998e-07 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863650+0000", 2024-09-16T13:18:13.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8479999999999999e-06 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863655+0000", 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863659+0000", 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.422e-06 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884422+0000", 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020762930999999998 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884429+0000", 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0199999999999997e-06 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35736 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863663+0000", 2024-09-16T13:18:13.221 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3220204600000001, 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020794782000000001, 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35736 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863663+0000", 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863663+0000", 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863663+0000", 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7599999999999998e-07 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863665+0000", 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9230000000000001e-06 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863670+0000", 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4129999999999997e-06 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863674+0000", 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884451+0000", 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020777711000000001 2024-09-16T13:18:13.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884457+0000", 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.02e-06 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35737 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863677+0000", 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.322005678, 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020816028, 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.225 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35737 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863677+0000", 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863677+0000", 2024-09-16T13:18:13.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863678+0000", 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863680+0000", 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9609999999999999e-06 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863685+0000", 2024-09-16T13:18:13.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4469999999999997e-06 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863694+0000", 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7509999999999997e-06 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884487+0000", 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020793345000000001 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884493+0000", 2024-09-16T13:18:13.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.127e-06 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35738 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863698+0000", 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.321985347, 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020825199999999999, 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35738 2024-09-16T13:18:13.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863698+0000", 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863698+0000", 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863698+0000", 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.03e-07 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863700+0000", 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863705+0000", 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3310000000000003e-06 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863709+0000", 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.552e-06 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884516+0000", 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020807651999999999 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884523+0000", 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3709999999999996e-06 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35739 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863713+0000", 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3219704849999996, 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020845658999999999, 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35739 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863713+0000", 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863713+0000", 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863713+0000", 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7099999999999997e-07 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863715+0000", 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9180000000000001e-06 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863721+0000", 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6899999999999997e-06 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863724+0000", 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884546+0000", 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020821507 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884558+0000", 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2660000000000001e-05 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35740 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863728+0000", 2024-09-16T13:18:13.236 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.321955472, 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020859386000000001, 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35740 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863728+0000", 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863728+0000", 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863728+0000", 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3500000000000002e-07 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863730+0000", 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:18:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863735+0000", 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2480000000000004e-06 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863738+0000", 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.585e-06 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884581+0000", 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020842362 2024-09-16T13:18:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884587+0000", 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0669999999999997e-06 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35741 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863743+0000", 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3219404370000003, 2024-09-16T13:18:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020873149000000001, 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35741 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863743+0000", 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863743+0000", 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863743+0000", 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863750+0000", 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863755+0000", 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863758+0000", 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3060000000000002e-06 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884610+0000", 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020851545999999999 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884616+0000", 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0360000000000003e-06 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35742 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863762+0000", 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3219205980000002, 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020882847999999999, 2024-09-16T13:18:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35742 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863762+0000", 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863762+0000", 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863763+0000", 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863765+0000", 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9719999999999999e-06 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863771+0000", 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2899999999999999e-06 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863774+0000", 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2449999999999998e-06 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884638+0000", 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020863711 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884645+0000", 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2370000000000003e-06 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35743 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863778+0000", 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3219048710000001, 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020895717000000001, 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35743 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863778+0000", 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863778+0000", 2024-09-16T13:18:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863779+0000", 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9400000000000001e-07 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863781+0000", 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1090000000000001e-06 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863786+0000", 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8119999999999996e-06 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863790+0000", 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8060000000000001e-06 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884668+0000", 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020877913000000001 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884674+0000", 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6829999999999996e-06 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35744 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863827+0000", 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3218556340000003, 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020880882, 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35744 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863827+0000", 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863827+0000", 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863828+0000", 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8400000000000003e-07 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863831+0000", 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8109999999999999e-06 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863840+0000", 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.9530000000000005e-06 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863844+0000", 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2259999999999999e-06 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884696+0000", 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020851919999999999 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884708+0000", 2024-09-16T13:18:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2187999999999999e-05 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35745 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863848+0000", 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3218348540000004, 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020889345, 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35745 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863848+0000", 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863848+0000", 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863849+0000", 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863851+0000", 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.035e-06 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863856+0000", 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7459999999999997e-06 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863860+0000", 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6370000000000001e-06 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884731+0000", 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020871137000000001 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884738+0000", 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2469999999999997e-06 2024-09-16T13:18:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35746 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863864+0000", 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3218190610000002, 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020908428, 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35746 2024-09-16T13:18:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863864+0000", 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863864+0000", 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863864+0000", 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863871+0000", 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.302e-06 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863877+0000", 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9460000000000003e-06 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863880+0000", 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.659e-06 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884762+0000", 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020881813999999999 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884772+0000", 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0349999999999999e-05 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:35747 3.b 3.eb7e580b (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:05.863884+0000", 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.3217986110000002, 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020917018999999999, 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "reached pg", 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 35747 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863884+0000", 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863884+0000", 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863885+0000", 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9299999999999999e-07 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863887+0000", 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9230000000000001e-06 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863892+0000", 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5670000000000002e-06 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.863896+0000", 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3919999999999999e-06 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884795+0000", 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.020899636999999999 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:05.884801+0000", 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1070000000000001e-06 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:13.268 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:13.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:13.269 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:13.271 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:13.281 INFO:teuthology.orchestra.run.smithi098.stdout:4514 trunc from 0x64cf0e2 to 0x7d65fce 2024-09-16T13:18:13.292 INFO:teuthology.orchestra.run.smithi098.stdout:4515 read 0x1675dd7 thru 0x167c5e3 (0x680d bytes) 2024-09-16T13:18:13.294 INFO:teuthology.orchestra.run.smithi098.stdout:4516 punch from 0x5385130 to 0x5386553, (0x1423 bytes) 2024-09-16T13:18:13.295 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:13.295 INFO:teuthology.orchestra.run.smithi098.stdout:4518 punch from 0x592cbe1 to 0x592cda5, (0x1c4 bytes) 2024-09-16T13:18:13.295 INFO:teuthology.orchestra.run.smithi098.stdout:4519 write 0x26cf5f1 thru 0x26d448a (0x4e9a bytes) 2024-09-16T13:18:13.317 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:13.311+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:13.378 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:13.372+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:13.381 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:13.381 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:13.382 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:13.382 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:13.391 INFO:teuthology.orchestra.run.smithi098.stdout:4521 write 0x85e964a thru 0x85f4915 (0xb2cc bytes) 2024-09-16T13:18:13.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:13.397 INFO:teuthology.orchestra.run.smithi098.stdout:4522 write 0x667d74e thru 0x6687b8a (0xa43d bytes) 2024-09-16T13:18:13.478 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:13.471+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:13.494 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:13.494 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:13.494 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:13.494 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:13.494 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout:4523 writesame 0xebccd95 thru 0xebd7aae data_size 0x1(0xad1a bytes) 2024-09-16T13:18:13.501 INFO:teuthology.orchestra.run.smithi098.stdout:4524 write 0xe9efff2 thru 0xe9f7422 (0x7431 bytes) 2024-09-16T13:18:13.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:13.527 INFO:teuthology.orchestra.run.smithi098.stdout:4525 trunc from 0xebd7aaf to 0xa63aa7 2024-09-16T13:18:13.580 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:13.571+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:13.680 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:13.671+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:13.685 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:13.685 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:13.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:13.685 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:13.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.685 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.685 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:13.685 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.47025685, 2024-09-16T13:18:13.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:18:13.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:18:13.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64676 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6717440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.558704+0000", 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.121336667, 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038895784000000003, 2024-09-16T13:18:13.690 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64676 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558704+0000", 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558704+0000", 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558704+0000", 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558706+0000", 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1160000000000002e-06 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558712+0000", 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4140000000000002e-06 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558715+0000", 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4539999999999999e-06 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.594853+0000", 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.036138116999999997 2024-09-16T13:18:13.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.594930+0000", 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6243999999999997e-05 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597600+0000", 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026700539999999998 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64686 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6799360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.694 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.558938+0000", 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.12110291099999999, 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038103766999999997, 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64686 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558938+0000", 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558938+0000", 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558938+0000", 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558940+0000", 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.136e-06 2024-09-16T13:18:13.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558945+0000", 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999997e-06 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558949+0000", 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5839999999999999e-06 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.596876+0000", 2024-09-16T13:18:13.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.03792645 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.596944+0000", 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8967999999999994e-05 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597041+0000", 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6922999999999993e-05 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64687 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6807552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.558953+0000", 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.12108791100000001, 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038272869000000001, 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64687 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558953+0000", 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558953+0000", 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558953+0000", 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9099999999999999e-07 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558955+0000", 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558960+0000", 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3140000000000003e-06 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558967+0000", 2024-09-16T13:18:13.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1770000000000003e-06 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597065+0000", 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038098672 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597132+0000", 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6657999999999998e-05 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597226+0000", 2024-09-16T13:18:13.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3665999999999994e-05 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64688 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6815744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.558971+0000", 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.12107003, 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038439464, 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.703 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64688 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558971+0000", 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558971+0000", 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558971+0000", 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5799999999999999e-07 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558973+0000", 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0729999999999999e-06 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558979+0000", 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4229999999999996e-06 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558982+0000", 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597246+0000", 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038264063000000001 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597312+0000", 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5974000000000002e-05 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597410+0000", 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7684999999999997e-05 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64689 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6823936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.558987+0000", 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.121053384, 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038595690000000002, 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.707 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64689 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558987+0000", 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558987+0000", 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558988+0000", 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558990+0000", 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.182e-06 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558995+0000", 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2639999999999999e-06 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559000+0000", 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8210000000000001e-06 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597431+0000", 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038431532999999997 2024-09-16T13:18:13.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597494+0000", 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2881999999999999e-05 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597583+0000", 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8653000000000007e-05 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64690 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6832128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559004+0000", 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.12103677, 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038788806000000002, 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64690 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.712 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559004+0000", 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559004+0000", 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559004+0000", 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559006+0000", 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.962e-06 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559012+0000", 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3990000000000003e-06 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559016+0000", 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3850000000000004e-06 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597629+0000", 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038612550000000002 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597696+0000", 2024-09-16T13:18:13.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7965000000000004e-05 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597793+0000", 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6099000000000003e-05 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64691 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6840320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559019+0000", 2024-09-16T13:18:13.716 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.121021088, 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040968721, 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64691 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559019+0000", 2024-09-16T13:18:13.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559019+0000", 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559020+0000", 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.53e-07 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559022+0000", 2024-09-16T13:18:13.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0999999999999998e-06 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559029+0000", 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.878e-06 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559036+0000", 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000003e-06 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597820+0000", 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038783718000000002 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.599886+0000", 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020662549999999999 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.599988+0000", 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101927 2024-09-16T13:18:13.720 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64692 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6848512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559054+0000", 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.12098625, 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041108762, 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64692 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559054+0000", 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559054+0000", 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559055+0000", 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5400000000000001e-07 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559057+0000", 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3769999999999999e-06 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559063+0000", 2024-09-16T13:18:13.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559067+0000", 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7230000000000002e-06 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600011+0000", 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040944095999999999 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600108+0000", 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6917999999999996e-05 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600163+0000", 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5012000000000001e-05 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64693 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6856704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.725 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559071+0000", 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.120969361, 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041219474999999998, 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64693 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559071+0000", 2024-09-16T13:18:13.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559071+0000", 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559072+0000", 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9900000000000001e-07 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559078+0000", 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559084+0000", 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7860000000000002e-06 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559087+0000", 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6459999999999999e-06 2024-09-16T13:18:13.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600176+0000", 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041088434 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600213+0000", 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7413000000000002e-05 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600291+0000", 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7542999999999999e-05 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64694 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6864896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559093+0000", 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.12094792, 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041365622999999997, 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.730 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64694 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559093+0000", 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559093+0000", 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559093+0000", 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2899999999999999e-07 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559095+0000", 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0980000000000001e-06 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559101+0000", 2024-09-16T13:18:13.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4240000000000001e-06 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559105+0000", 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4989999999999996e-06 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600311+0000", 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041206185999999999 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5061999999999995e-05 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600458+0000", 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1924999999999999e-05 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.734 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64695 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6873088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559110+0000", 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.120930225, 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041445530000000001, 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64695 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559110+0000", 2024-09-16T13:18:13.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559110+0000", 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559111+0000", 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9799999999999999e-07 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559113+0000", 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559460+0000", 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000347022 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559469+0000", 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8699999999999998e-06 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600471+0000", 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041002200000000003 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600506+0000", 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5426999999999998e-05 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600556+0000", 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9669999999999997e-05 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.738 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64696 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6881280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559476+0000", 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.120564617, 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041184922999999998, 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64696 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559476+0000", 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559476+0000", 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559477+0000", 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4700000000000001e-07 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559481+0000", 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0389999999999998e-06 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559489+0000", 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2169999999999994e-06 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559493+0000", 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9770000000000002e-06 2024-09-16T13:18:13.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600568+0000", 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041075118000000001 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600606+0000", 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7793999999999998e-05 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600661+0000", 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5031000000000002e-05 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64697 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6889472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559498+0000", 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.120542571, 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041268907000000001, 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64697 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559498+0000", 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559498+0000", 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562851+0000", 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033532179999999998 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562858+0000", 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7440000000000001e-06 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562871+0000", 2024-09-16T13:18:13.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3251000000000001e-05 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562878+0000", 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.832e-06 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600673+0000", 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037795424000000001 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600709+0000", 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5457000000000003e-05 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600767+0000", 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7980999999999999e-05 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64698 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6897664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562885+0000", 2024-09-16T13:18:13.747 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.11715548200000001, 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037978407999999998, 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64698 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562885+0000", 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562885+0000", 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562886+0000", 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5899999999999995e-07 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562889+0000", 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2459999999999999e-06 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562897+0000", 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3129999999999999e-06 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562902+0000", 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600780+0000", 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037878214 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600815+0000", 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5512999999999997e-05 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600863+0000", 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8226999999999998e-05 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.751 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64699 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6905856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562907+0000", 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.11713347, 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038061630999999999, 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64699 2024-09-16T13:18:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562907+0000", 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562907+0000", 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562908+0000", 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562910+0000", 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6570000000000001e-06 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562916+0000", 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2299999999999996e-06 2024-09-16T13:18:13.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562920+0000", 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037956075999999998 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600918+0000", 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1332999999999999e-05 2024-09-16T13:18:13.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600969+0000", 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0976000000000001e-05 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64700 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6914048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562924+0000", 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.11711648299999999, 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038145167000000001, 2024-09-16T13:18:13.756 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64700 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562924+0000", 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562924+0000", 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562925+0000", 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562927+0000", 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1629999999999999e-06 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562947+0000", 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9930000000000001e-05 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562951+0000", 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.403e-06 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600981+0000", 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038029962 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601018+0000", 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7503000000000003e-05 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601069+0000", 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0764999999999999e-05 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64701 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6922240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562956+0000", 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.11708468399999999, 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038235064999999999, 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64701 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562956+0000", 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562956+0000", 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562957+0000", 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:18:13.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562959+0000", 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5330000000000001e-06 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562965+0000", 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2349999999999996e-06 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562969+0000", 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8739999999999997e-06 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601082+0000", 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038112372999999998 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601124+0000", 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2743000000000001e-05 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601191+0000", 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6612000000000005e-05 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64702 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6930432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562973+0000", 2024-09-16T13:18:13.765 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.117067486, 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038316650000000001, 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64702 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562973+0000", 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562973+0000", 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562974+0000", 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9800000000000004e-07 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562976+0000", 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1569999999999998e-06 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562981+0000", 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562985+0000", 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601204+0000", 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038218528000000002 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601240+0000", 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6000000000000001e-05 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601290+0000", 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9731000000000002e-05 2024-09-16T13:18:13.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64703 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6938624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562989+0000", 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.117051107, 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038408454000000002, 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:13.770 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64703 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562989+0000", 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562989+0000", 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562990+0000", 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562992+0000", 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1639999999999999e-06 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562998+0000", 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.711e-06 2024-09-16T13:18:13.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.563001+0000", 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601303+0000", 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038301833 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601339+0000", 2024-09-16T13:18:13.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5377999999999999e-05 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601398+0000", 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9281999999999999e-05 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:13.774 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:13.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:13.781 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:13.776+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:13.846 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:13.856 INFO:teuthology.orchestra.run.smithi098.stdout:4527 trunc from 0xa63aa7 to 0x4bd961a 2024-09-16T13:18:13.867 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:13.868 INFO:teuthology.orchestra.run.smithi098.stdout:4529 write 0xcda87e thru 0xce7d5c (0xd4df bytes) 2024-09-16T13:18:13.882 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:13.876+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:13.884 INFO:teuthology.orchestra.run.smithi098.stdout:4532 trunc from 0x4bd961a to 0x2ad399 2024-09-16T13:18:13.914 INFO:teuthology.orchestra.run.smithi098.stdout:4533 read 0x21bbd7 thru 0x22b8c5 (0xfcef bytes) 2024-09-16T13:18:13.915 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:13.915 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:13.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:13.935 INFO:teuthology.orchestra.run.smithi098.stdout:4534 write 0x433ec99 thru 0x434b557 (0xc8bf bytes) 2024-09-16T13:18:13.964 INFO:teuthology.orchestra.run.smithi098.stdout:4536 write 0x9f663c5 thru 0x9f71f2c (0xbb68 bytes) 2024-09-16T13:18:13.983 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:13.980+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:13.985 INFO:teuthology.orchestra.run.smithi098.stdout:4537 trunc from 0x9f71f2d to 0x3fe7276 2024-09-16T13:18:14.023 INFO:teuthology.orchestra.run.smithi098.stdout:4540 trunc from 0x3fe7276 to 0xdb60b16 2024-09-16T13:18:14.023 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:14.024 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:14.024 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:14.024 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:14.024 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:14.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:14.033 INFO:teuthology.orchestra.run.smithi098.stdout:4541 write 0x3ebe6d2 thru 0x3ebfd54 (0x1683 bytes) 2024-09-16T13:18:14.039 INFO:teuthology.orchestra.run.smithi098.stdout:4542 writesame 0xc18eb6a thru 0xc19aa1f data_size 0x67(0xbeb6 bytes) 2024-09-16T13:18:14.045 INFO:teuthology.orchestra.run.smithi098.stdout:4543 write 0x554c342 thru 0x5553138 (0x6df7 bytes) 2024-09-16T13:18:14.048 INFO:teuthology.orchestra.run.smithi098.stdout:4544 writesame 0x5a71fc thru 0x5afdd0 data_size 0x1(0x8bd5 bytes) 2024-09-16T13:18:14.083 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:14.075+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:14.102 INFO:teuthology.orchestra.run.smithi098.stdout:4545 write 0xeaca7f1 thru 0xeacb522 (0xd32 bytes) 2024-09-16T13:18:14.108 INFO:teuthology.orchestra.run.smithi098.stdout:4546 write 0xb4e3aee thru 0xb4eef70 (0xb483 bytes) 2024-09-16T13:18:14.124 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2525 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 557056~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094901938999996, 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014602618, 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2525 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024702+0000", 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024705+0000", 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4650000000000001e-06 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024711+0000", 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024716+0000", 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1710000000000004e-06 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039096+0000", 2024-09-16T13:18:14.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014380184000000001 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039187+0000", 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1065999999999999e-05 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039304+0000", 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117333 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.129 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2526 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1507328~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094883769999996, 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014823846, 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2526 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.130 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024720+0000", 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024722+0000", 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2800000000000002e-06 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024729+0000", 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1720000000000004e-06 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024733+0000", 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2560000000000004e-06 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039337+0000", 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014604026000000001 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039427+0000", 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0437000000000002e-05 2024-09-16T13:18:14.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039544+0000", 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116234 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2527 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1605632~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094866006999993, 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015041116, 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.134 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2527 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024737+0000", 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024738+0000", 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024746+0000", 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2130000000000008e-06 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024753+0000", 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024758+0000", 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7670000000000003e-06 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039578+0000", 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014820533 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039670+0000", 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1825999999999998e-05 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039779+0000", 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108424 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2528 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094840568999999, 2024-09-16T13:18:14.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015245242000000001, 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2528 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024763+0000", 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024765+0000", 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1040000000000001e-06 2024-09-16T13:18:14.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024772+0000", 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024776+0000", 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039810+0000", 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015034194000000001 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.039897+0000", 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6864e-05 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040008+0000", 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110802 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2529 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2555904~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094821449999998, 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015471574, 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2529 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.143 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024782+0000", 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5499999999999998e-07 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024785+0000", 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5969999999999999e-06 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024800+0000", 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4681999999999999e-05 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024806+0000", 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6749999999999996e-06 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040039+0000", 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015233007999999999 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040143+0000", 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010322799999999999 2024-09-16T13:18:14.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040254+0000", 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011092899999999999 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2530 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2588672~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094791169999993, 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015667606000000001, 2024-09-16T13:18:14.147 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2530 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024812+0000", 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024813+0000", 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024815+0000", 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5160000000000001e-06 2024-09-16T13:18:14.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024827+0000", 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1915e-05 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024834+0000", 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5180000000000001e-06 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040286+0000", 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015452565999999999 2024-09-16T13:18:14.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040375+0000", 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8159999999999996e-05 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040480+0000", 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105388 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.151 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2531 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3538944~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094763955000005, 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015867612999999999, 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2531 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024840+0000", 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1600000000000001e-07 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024843+0000", 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6960000000000001e-06 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024856+0000", 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-05 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024863+0000", 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6069999999999996e-06 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040513+0000", 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015650662999999999 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040599+0000", 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5439999999999995e-05 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040707+0000", 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108461 2024-09-16T13:18:14.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2532 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094733450000007, 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061912000000001, 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.156 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2532 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024870+0000", 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024871+0000", 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024874+0000", 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5480000000000002e-06 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024881+0000", 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0570000000000004e-06 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024886+0000", 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040742+0000", 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015856808 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040824+0000", 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1087000000000005e-05 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040932+0000", 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108378 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2533 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 589824~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094712924999996, 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016272282999999998, 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2533 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:18:14.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024891+0000", 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024893+0000", 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419999999999999e-06 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024900+0000", 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3289999999999999e-06 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024903+0000", 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.040965+0000", 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016061565999999999 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041052+0000", 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7213000000000007e-05 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041163+0000", 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011066899999999999 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2534 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 3604480~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:18:14.164 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094694394000001, 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016482746999999999, 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2534 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:18:14.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024909+0000", 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024910+0000", 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1299999999999999e-07 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024912+0000", 2024-09-16T13:18:14.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024920+0000", 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7379999999999994e-06 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024924+0000", 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.036e-06 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041197+0000", 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016273242 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041282+0000", 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5013999999999996e-05 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041392+0000", 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109567 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.168 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2535 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 688128~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094673732000004, 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016691902000000002, 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2535 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.169 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024930+0000", 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7599999999999997e-07 2024-09-16T13:18:14.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024932+0000", 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024949+0000", 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7053000000000001e-05 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024961+0000", 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041425+0000", 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016463851000000002 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041515+0000", 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9549999999999995e-05 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041622+0000", 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107132 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2536 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1638400~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094635925999995, 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016880869999999999, 2024-09-16T13:18:14.173 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2536 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024968+0000", 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024971+0000", 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.615e-06 2024-09-16T13:18:14.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024988+0000", 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6912999999999999e-05 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024993+0000", 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041653+0000", 2024-09-16T13:18:14.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016660245000000001 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041743+0000", 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0099999999999995e-05 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041848+0000", 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105404 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.177 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2537 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 1671168~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094605173000005, 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017075346000000002, 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2537 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.178 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024998+0000", 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.024999+0000", 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9200000000000001e-07 2024-09-16T13:18:14.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025002+0000", 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0060000000000001e-06 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025009+0000", 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025013+0000", 2024-09-16T13:18:14.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1840000000000001e-06 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041886+0000", 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016873249999999999 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.041971+0000", 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4487000000000006e-05 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042074+0000", 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102847 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2538 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2621440~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094585002000002, 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017314057000000001, 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.182 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2538 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025018+0000", 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025019+0000", 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025021+0000", 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3989999999999998e-06 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025033+0000", 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1985e-05 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025043+0000", 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3810000000000005e-06 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042107+0000", 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017064471000000001 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042203+0000", 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5810999999999995e-05 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042333+0000", 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012956099999999999 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:2539 3.c 3:35cd0a46:::rbd_data.112de98401a1.0000000000000001:4 [read 2654208~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.094557273999996, 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017500180000000001, 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 2539 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025046+0000", 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025047+0000", 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025053+0000", 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025058+0000", 2024-09-16T13:18:14.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.025061+0000", 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.072e-06 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042365+0000", 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017304885999999998 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042458+0000", 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2991000000000005e-05 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:04.042546+0000", 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7904000000000004e-05 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.190 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:22787 3.17 3:edd0f01e:::rbd_object_map.112db5030c67:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.802172691999999, 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015269972, 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 22787 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317431+0000", 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317432+0000", 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.127e-06 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317439+0000", 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0500000000000003e-06 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317458+0000", 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9326000000000001e-05 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.317467+0000", 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0669999999999998e-06 2024-09-16T13:18:14.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329243+0000", 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011775551 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329293+0000", 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9709000000000001e-05 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.329505+0000", 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021237200000000001 2024-09-16T13:18:14.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.330969+0000", 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014635290000000001 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332590+0000", 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001621485 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332655+0000", 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4670000000000003e-05 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:41.332701+0000", 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6085999999999999e-05 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:23429 3.5 3:a41c7f85:::rbd_object_map.112dc49cabe2:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.030476525000001, 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014415534000000001, 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.196 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 23429 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089127+0000", 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089128+0000", 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7599999999999996e-07 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089140+0000", 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1939999999999999e-05 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089164+0000", 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4430000000000002e-05 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089173+0000", 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8200000000000003e-06 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.097878+0000", 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087055320000000002 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.098035+0000", 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156537 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.098605+0000", 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057031900000000001 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103250+0000", 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046448119999999999 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103351+0000", 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100763 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103472+0000", 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121091 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103542+0000", 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0513999999999996e-05 2024-09-16T13:18:14.201 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:23430 3.5 3:a41c7f85:::rbd_object_map.112dc49cabe2:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.030420634999999, 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016212550999999999, 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.202 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.203 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 23430 2024-09-16T13:18:14.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.203 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:18:14.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089183+0000", 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089184+0000", 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.33e-07 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089192+0000", 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8730000000000007e-06 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089210+0000", 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8127000000000001e-05 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.089217+0000", 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7130000000000005e-06 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.099775+0000", 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010557957999999999 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.099894+0000", 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011883999999999999 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.100458+0000", 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056368200000000003 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.103330+0000", 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028723690000000001 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105122+0000", 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017912430000000001 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105297+0000", 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175055 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:17:46.105395+0000", 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8758000000000006e-05 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.208 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.297132251000001, 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.209 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:18:14.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:14.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:63500 3.d 3:b570719d:::rbd_data.112d338bfe8e.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 6805326~42045] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:14.215 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.445693+0000", 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.67391052500000004, 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.031481095000000001, 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63500 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.445693+0000", 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.445693+0000", 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.445695+0000", 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.463625+0000", 2024-09-16T13:18:14.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017929852999999999 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.463663+0000", 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7842999999999997e-05 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.463676+0000", 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3155e-05 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.463765+0000", 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9203999999999995e-05 2024-09-16T13:18:14.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.463988+0000", 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022266199999999999 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.477017+0000", 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013028988 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.477086+0000", 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8684999999999997e-05 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.477174+0000", 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8475999999999998e-05 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.220 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:14.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:18:14.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:14.223 INFO:teuthology.orchestra.run.smithi098.stdout:4548 trunc from 0xeacb523 to 0x99c3813 2024-09-16T13:18:14.223 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:14.223 INFO:teuthology.orchestra.run.smithi098.stdout:4551 read 0x6e7c25b thru 0x6e7ed6c (0x2b12 bytes) 2024-09-16T13:18:14.224 INFO:teuthology.orchestra.run.smithi098.stdout:4552 write 0xbca7898 thru 0xbcb4b16 (0xd27f bytes) 2024-09-16T13:18:14.224 INFO:teuthology.orchestra.run.smithi098.stdout:4553 read 0x4431555 thru 0x4434896 (0x3342 bytes) 2024-09-16T13:18:14.224 INFO:teuthology.orchestra.run.smithi098.stdout:4554 punch from 0x9fad29a to 0x9fb24ac, (0x5212 bytes) 2024-09-16T13:18:14.224 INFO:teuthology.orchestra.run.smithi098.stdout:4555 write 0x23c9740 thru 0x23d7cce (0xe58f bytes) 2024-09-16T13:18:14.224 INFO:teuthology.orchestra.run.smithi098.stdout:4556 read 0x94b21be thru 0x94b916f (0x6fb2 bytes) 2024-09-16T13:18:14.224 INFO:teuthology.orchestra.run.smithi098.stdout:4557 write 0xc19a2ca thru 0xc19d592 (0x32c9 bytes) 2024-09-16T13:18:14.224 INFO:teuthology.orchestra.run.smithi098.stdout:4560 trunc from 0xc19d593 to 0xa822933 2024-09-16T13:18:14.224 INFO:teuthology.orchestra.run.smithi098.stdout:4561 write 0x344e8e7 thru 0x3459611 (0xad2b bytes) 2024-09-16T13:18:14.224 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:14.216+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:14.239 INFO:teuthology.orchestra.run.smithi098.stdout:4563 trunc from 0xa822933 to 0x366ef2 2024-09-16T13:18:14.305 INFO:teuthology.orchestra.run.smithi098.stdout:4564 write 0xb7c17ac thru 0xb7cdb8b (0xc3e0 bytes) 2024-09-16T13:18:14.312 INFO:teuthology.orchestra.run.smithi098.stdout:4565 write 0x6e7287 thru 0x6e84db (0x1255 bytes) 2024-09-16T13:18:14.317 INFO:teuthology.orchestra.run.smithi098.stdout:4566 read 0xa2ccd2f thru 0xa2d9f90 (0xd262 bytes) 2024-09-16T13:18:14.317 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:14.317 INFO:teuthology.orchestra.run.smithi098.stdout:4569 read 0x58f39f3 thru 0x5902e31 (0xf43f bytes) 2024-09-16T13:18:14.317 INFO:teuthology.orchestra.run.smithi098.stdout:4570 punch from 0x88be82 to 0x8902ca, (0x4448 bytes) 2024-09-16T13:18:14.317 INFO:teuthology.orchestra.run.smithi098.stdout:4572 write 0x4dbf358 thru 0x4dc3f9c (0x4c45 bytes) 2024-09-16T13:18:14.322 INFO:teuthology.orchestra.run.smithi098.stdout:4573 write 0x47df782 thru 0x47e6fde (0x785d bytes) 2024-09-16T13:18:14.323 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:14.315+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:14.327 INFO:teuthology.orchestra.run.smithi098.stdout:4575 writesame 0x42dc680 thru 0x42df652 data_size 0x21(0x2fd3 bytes) 2024-09-16T13:18:14.332 INFO:teuthology.orchestra.run.smithi098.stdout:4576 read 0x9ce7617 thru 0x9cf0b24 (0x950e bytes) 2024-09-16T13:18:14.334 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:14.334 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:14.342 INFO:teuthology.orchestra.run.smithi151.stdout:3.f 2024-09-16T13:18:14.342 INFO:teuthology.orchestra.run.smithi151.stdout:3.e 2024-09-16T13:18:14.342 INFO:teuthology.orchestra.run.smithi151.stdout:3.d 2024-09-16T13:18:14.342 INFO:teuthology.orchestra.run.smithi151.stdout:3.7 2024-09-16T13:18:14.342 INFO:teuthology.orchestra.run.smithi151.stdout:3.16 2024-09-16T13:18:14.342 INFO:teuthology.orchestra.run.smithi151.stdout:3.13 2024-09-16T13:18:14.342 INFO:teuthology.orchestra.run.smithi151.stdout:3.12 2024-09-16T13:18:14.342 INFO:teuthology.orchestra.run.smithi151.stdout:3.11 2024-09-16T13:18:14.343 INFO:teuthology.orchestra.run.smithi151.stdout:3.15 2024-09-16T13:18:14.343 INFO:teuthology.orchestra.run.smithi151.stdout:2.7 2024-09-16T13:18:14.343 INFO:teuthology.orchestra.run.smithi151.stdout:2.5 2024-09-16T13:18:14.343 INFO:teuthology.orchestra.run.smithi151.stdout:3.1b 2024-09-16T13:18:14.343 INFO:teuthology.orchestra.run.smithi151.stdout:3.14 2024-09-16T13:18:14.343 INFO:teuthology.orchestra.run.smithi151.stdout:2.0 2024-09-16T13:18:14.343 INFO:teuthology.orchestra.run.smithi151.stdout:1.0 2024-09-16T13:18:14.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:14.347 INFO:teuthology.orchestra.run.smithi098.stdout:4577 writesame 0xdbf941a thru 0xdc07171 data_size 0x24(0xdd58 bytes) 2024-09-16T13:18:14.349 INFO:teuthology.orchestra.run.smithi098.stdout:4578 trunc from 0xdc07172 to 0xc615777 2024-09-16T13:18:14.360 INFO:teuthology.orchestra.run.smithi098.stdout:4579 write 0x91c5d96 thru 0x91d3cf1 (0xdf5c bytes) 2024-09-16T13:18:14.367 INFO:teuthology.orchestra.run.smithi098.stdout:4580 punch from 0x8379c98 to 0x837e947, (0x4caf bytes) 2024-09-16T13:18:14.367 INFO:teuthology.orchestra.run.smithi098.stdout:4581 punch from 0x5ba7ce6 to 0x5baf446, (0x7760 bytes) 2024-09-16T13:18:14.367 INFO:teuthology.orchestra.run.smithi098.stdout:4582 write 0x8d935c2 thru 0x8d94285 (0xcc4 bytes) 2024-09-16T13:18:14.396 INFO:teuthology.orchestra.run.smithi098.stdout:4583 trunc from 0xc615777 to 0x1ab902 2024-09-16T13:18:14.424 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:14.415+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:14.445 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:14.445 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:14.453 INFO:teuthology.orchestra.run.smithi098.stdout:4584 trunc from 0x1ab902 to 0x96f0e67 2024-09-16T13:18:14.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:14.463 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:14.473 INFO:teuthology.orchestra.run.smithi098.stdout:4586 write 0x98829c6 thru 0x98868c0 (0x3efb bytes) 2024-09-16T13:18:14.481 INFO:teuthology.orchestra.run.smithi098.stdout:4587 punch from 0xfb5e36 to 0xfbe152, (0x831c bytes) 2024-09-16T13:18:14.481 INFO:teuthology.orchestra.run.smithi098.stdout:4589 read 0x508bd0f thru 0x508cd65 (0x1057 bytes) 2024-09-16T13:18:14.481 INFO:teuthology.orchestra.run.smithi098.stdout:4590 punch from 0x2d86508 to 0x2d88b8c, (0x2684 bytes) 2024-09-16T13:18:14.481 INFO:teuthology.orchestra.run.smithi098.stdout:4591 write 0x41ba15c thru 0x41c0bd2 (0x6a77 bytes) 2024-09-16T13:18:14.488 INFO:teuthology.orchestra.run.smithi098.stdout:4592 punch from 0x27a166a to 0x27a2eb2, (0x1848 bytes) 2024-09-16T13:18:14.488 INFO:teuthology.orchestra.run.smithi098.stdout:4593 writesame 0x48a268a thru 0x48b1de6 data_size 0x19(0xf75d bytes) 2024-09-16T13:18:14.508 INFO:teuthology.orchestra.run.smithi098.stdout:4594 trunc from 0x98868c1 to 0x3d617a6 2024-09-16T13:18:14.524 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:14.515+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:14.560 INFO:teuthology.orchestra.run.smithi098.stdout:4595 write 0xc754a86 thru 0xc764946 (0xfec1 bytes) 2024-09-16T13:18:14.584 INFO:teuthology.orchestra.run.smithi098.stdout:4596 trunc from 0xc764947 to 0x35d786a 2024-09-16T13:18:14.608 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:14.608 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:14.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:14.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:14.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.608 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.608 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:14.608 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.722078174, 2024-09-16T13:18:14.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:18:14.608 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:18:14.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.612 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7220593389999994, 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:14.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:18:14.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7220402000000004, 2024-09-16T13:18:14.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:18:14.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:18:14.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:18:14.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.625 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7220231419999994, 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7220057820000001, 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:14.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7219880730000003, 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:18:14.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:18:14.638 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7219707740000008, 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:18:14.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7219356759999993, 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.643 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7219125210000001, 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:18:14.647 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.651 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7218715020000008, 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:14.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.655 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7218486019999997, 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.656 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:18:14.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:18:14.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7218235530000001, 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.660 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:14.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:18:14.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:14.664 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7217983320000005, 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:18:14.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:18:14.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7217799669999998, 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:18:14.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7215168300000006, 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.673 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:18:14.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.677 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7214918220000008, 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:18:14.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:18:14.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7214634899999997, 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.682 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:18:14.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7211747919999993, 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.686 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:18:14.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7211546819999999, 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:18:14.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.7169537580000007, 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:14.695 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:14.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:14.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:14.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:18:14.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:18:14.699 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.699 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.699 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.699 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:14.699 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:14.699 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:14.701 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:14.701 INFO:teuthology.orchestra.run.smithi098.stdout:4598 trunc from 0x35d786a to 0xd40c466 2024-09-16T13:18:14.701 INFO:teuthology.orchestra.run.smithi098.stdout:4599 writesame 0x38d9507 thru 0x38e7c94 data_size 0x2(0xe78e bytes) 2024-09-16T13:18:14.701 INFO:teuthology.orchestra.run.smithi098.stdout:4601 writesame 0xddc8566 thru 0xddcecf5 data_size 0x8(0x6790 bytes) 2024-09-16T13:18:14.701 INFO:teuthology.orchestra.run.smithi098.stdout:4602 clone 32 order 24 su 131072 sc 14 2024-09-16T13:18:14.702 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:14.691+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:14.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:14.802 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:14.799+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:14.809 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:14.809 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:14.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:14.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 3.15 --file /var/log/ceph/exp.3.15.7 2024-09-16T13:18:14.902 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:14.896+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:14.916 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:14.917 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:14.917 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:14.917 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:14.917 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:14.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:15.004 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:14.995+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:15.033 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66873 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4034560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4125457779999999, 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034635435999999999, 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.034 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66873 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:15.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615785+0000", 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615786+0000", 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615792+0000", 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1900000000000003e-06 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615795+0000", 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9450000000000002e-06 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650232+0000", 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034437669999999997 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650301+0000", 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8329000000000003e-05 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650420+0000", 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119132 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66874 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4042752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:15.038 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.412512572, 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034827871000000003, 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66874 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615820+0000", 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.063e-06 2024-09-16T13:18:15.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615826+0000", 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9739999999999999e-06 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615835+0000", 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5920000000000001e-06 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650445+0000", 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034610100999999997 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650512+0000", 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650645+0000", 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013316399999999999 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66875 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4050944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.412491663, 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032338000000003, 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66875 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615839+0000", 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:15.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615841+0000", 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9250000000000002e-06 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615846+0000", 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.711e-06 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615851+0000", 2024-09-16T13:18:15.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9579999999999998e-06 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650676+0000", 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034824524000000003 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650758+0000", 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1547999999999999e-05 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650871+0000", 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113244 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66876 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4059136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.412474768, 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035239591000000001, 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.047 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66876 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615856+0000", 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615858+0000", 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.857e-06 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615863+0000", 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5389999999999998e-06 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615866+0000", 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650899+0000", 2024-09-16T13:18:15.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032448000000001 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650973+0000", 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3627000000000005e-05 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651095+0000", 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012239400000000001 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66877 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4067328~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4124600680000001, 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035445691000000001, 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66877 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.052 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615872+0000", 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.703e-06 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615877+0000", 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2650000000000004e-06 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615881+0000", 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.292e-06 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651123+0000", 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035242373 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651198+0000", 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5096000000000001e-05 2024-09-16T13:18:15.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651316+0000", 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011757 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66878 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4075520~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4124461749999999, 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035647261999999999, 2024-09-16T13:18:15.056 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66878 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5400000000000002e-07 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615890+0000", 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6189999999999999e-06 2024-09-16T13:18:15.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615896+0000", 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6529999999999999e-06 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615900+0000", 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0069999999999999e-06 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651337+0000", 2024-09-16T13:18:15.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035437877999999999 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651395+0000", 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7219999999999998e-05 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651531+0000", 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013653100000000001 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66879 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4124268340000001, 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035838241999999999, 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66879 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.061 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615904+0000", 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615906+0000", 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615911+0000", 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6679999999999998e-06 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615914+0000", 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.151e-06 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651558+0000", 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035643672000000001 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651624+0000", 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6070999999999998e-05 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651742+0000", 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011729499999999999 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66880 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4091904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4124122539999999, 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036042148000000003, 2024-09-16T13:18:15.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66880 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2099999999999997e-07 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615924+0000", 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:18:15.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615930+0000", 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6250000000000004e-06 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615933+0000", 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3950000000000001e-06 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651773+0000", 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035839579000000003 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651846+0000", 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2602999999999996e-05 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651960+0000", 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114319 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.069 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66881 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4100096~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.412391237, 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036242563999999998, 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66881 2024-09-16T13:18:15.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615941+0000", 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.72e-06 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615946+0000", 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.259e-06 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615950+0000", 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651985+0000", 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036035821000000003 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652058+0000", 2024-09-16T13:18:15.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2651999999999995e-05 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652181+0000", 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123428 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66882 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4108288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.412377207, 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036437628999999999, 2024-09-16T13:18:15.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66882 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7300000000000002e-07 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615955+0000", 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8199999999999999e-06 2024-09-16T13:18:15.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615961+0000", 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2349999999999996e-06 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615965+0000", 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4620000000000001e-06 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652207+0000", 2024-09-16T13:18:15.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036241873000000001 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652276+0000", 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9105999999999999e-05 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652391+0000", 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011476000000000001 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66883 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4116480~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4123619839999999, 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036642620000000001, 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66883 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.77e-07 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615970+0000", 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7519999999999999e-06 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615977+0000", 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9349999999999997e-06 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615985+0000", 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1389999999999995e-06 2024-09-16T13:18:15.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652416+0000", 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036430381999999997 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652487+0000", 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1698000000000004e-05 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652611+0000", 2024-09-16T13:18:15.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123337 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66884 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4124672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4120975570000001, 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036613363000000003, 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.083 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66884 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.16e-07 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616237+0000", 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5719999999999999e-06 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616245+0000", 2024-09-16T13:18:15.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8820000000000001e-06 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616251+0000", 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9630000000000003e-06 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652638+0000", 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036387705999999999 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652719+0000", 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0469999999999994e-05 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652846+0000", 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126954 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66885 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4132864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:15.087 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4120750049999999, 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036785929000000002, 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66885 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:15.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0999999999999999e-07 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616258+0000", 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0269999999999998e-06 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616263+0000", 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8760000000000002e-06 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616267+0000", 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652873+0000", 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036606376000000003 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652956+0000", 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2824000000000005e-05 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653041+0000", 2024-09-16T13:18:15.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4918000000000002e-05 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66886 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4141056~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.412058576, 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078279999999998, 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.092 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66886 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616274+0000", 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.73e-06 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616279+0000", 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616283+0000", 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1420000000000002e-06 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653119+0000", 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036836016999999999 2024-09-16T13:18:15.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653204+0000", 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5575999999999995e-05 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653350+0000", 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014569499999999999 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66887 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4149248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4120440730000001, 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037308557999999999, 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66887 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:15.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.84e-07 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616288+0000", 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7099999999999999e-06 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616294+0000", 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4029999999999998e-06 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616297+0000", 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0079999999999998e-06 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653375+0000", 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078883 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653455+0000", 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0034e-05 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653595+0000", 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013913599999999999 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66888 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4157440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4120297019999999, 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537323999999997, 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.101 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66888 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616301+0000", 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616302+0000", 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616308+0000", 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616317+0000", 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2229999999999996e-06 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653620+0000", 2024-09-16T13:18:15.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037302572999999999 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653701+0000", 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1771e-05 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653838+0000", 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013640599999999999 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.105 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66889 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4165632~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.412009174, 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037755019000000001, 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66889 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.106 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2300000000000002e-07 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616323+0000", 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.905e-06 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616329+0000", 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4639999999999997e-06 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616332+0000", 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0180000000000002e-06 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653862+0000", 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037529973000000001 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653937+0000", 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4954000000000001e-05 2024-09-16T13:18:15.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654076+0000", 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013928200000000001 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66890 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4173824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4119937679999999, 2024-09-16T13:18:15.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037917073000000003, 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66890 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616337+0000", 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616338+0000", 2024-09-16T13:18:15.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616371+0000", 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2806999999999999e-05 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616375+0000", 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.827e-06 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654115+0000", 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.03773986 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654181+0000", 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5652000000000001e-05 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654253+0000", 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2789999999999999e-05 2024-09-16T13:18:15.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66891 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4182016~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4119458970000001, 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038160044999999997, 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.115 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66891 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616385+0000", 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616387+0000", 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.356e-06 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616393+0000", 2024-09-16T13:18:15.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9519999999999999e-06 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616396+0000", 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2959999999999999e-06 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654302+0000", 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037905031999999998 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654395+0000", 2024-09-16T13:18:15.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3560000000000005e-05 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654544+0000", 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014916500000000001 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66892 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4190208~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.4119300859999999, 2024-09-16T13:18:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038342319999999999, 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66892 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616407+0000", 2024-09-16T13:18:15.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4060000000000002e-06 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616412+0000", 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-06 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616419+0000", 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.037e-06 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654571+0000", 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038152835000000003 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654625+0000", 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3643000000000001e-05 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654742+0000", 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117348 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.124 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.124 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:15.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:15.126 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:15.120+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:15.218 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:18:15.225 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:15.220+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:15.240 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:15.240 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:15.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:15.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:15.326 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:15.320+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:15.346 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:15.346 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:15.347 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:15.347 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:15.347 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:15.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:15.427 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:15.424+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:15.448 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:15.460 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:15.460 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:15.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:15.460 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:15.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.460 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64860 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3899392~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.8919465500000001, 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035719773000000003, 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64860 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563336+0000", 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563342+0000", 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3240000000000002e-06 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563345+0000", 2024-09-16T13:18:15.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.598973+0000", 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035627631999999999 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599006+0000", 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3495999999999999e-05 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599054+0000", 2024-09-16T13:18:15.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7460999999999998e-05 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64861 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3907584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.8919316689999999, 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035815461, 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64861 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563351+0000", 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.854e-06 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563356+0000", 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5940000000000001e-06 2024-09-16T13:18:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563360+0000", 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4060000000000001e-06 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599076+0000", 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035716171999999997 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599116+0000", 2024-09-16T13:18:15.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9777999999999998e-05 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599164+0000", 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8307999999999999e-05 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64862 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.8919170759999999, 2024-09-16T13:18:15.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035954097999999997, 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64862 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563364+0000", 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563365+0000", 2024-09-16T13:18:15.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.821e-06 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563371+0000", 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0769999999999998e-06 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563374+0000", 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3560000000000001e-06 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599187+0000", 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035813153 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599252+0000", 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4873e-05 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599317+0000", 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5438000000000006e-05 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64863 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3923968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.8918915219999999, 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036051842000000001, 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64863 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.474 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:18:15.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563392+0000", 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3140000000000002e-06 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563398+0000", 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.066e-06 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563402+0000", 2024-09-16T13:18:15.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599339+0000", 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035937678000000001 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599377+0000", 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599441+0000", 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3380999999999995e-05 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64864 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3932160~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.891875118, 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036187015000000003, 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64864 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:15.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563406+0000", 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563411+0000", 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563417+0000", 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563421+0000", 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599458+0000", 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036036870999999998 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599515+0000", 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7067000000000001e-05 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599592+0000", 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7457000000000001e-05 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.482 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64865 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3940352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.8918557279999999, 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036265720000000001, 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64865 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563427+0000", 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.001e-06 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563433+0000", 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5489999999999997e-06 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563436+0000", 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.484e-06 2024-09-16T13:18:15.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599613+0000", 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036176815000000001 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599646+0000", 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3192999999999999e-05 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599690+0000", 2024-09-16T13:18:15.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4240000000000003e-05 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64866 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3948544~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.89183995, 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036355749999999999, 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.487 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64866 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563441+0000", 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563443+0000", 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0870000000000001e-06 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563448+0000", 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.062e-06 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563451+0000", 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599712+0000", 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036261007999999997 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599753+0000", 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0476999999999999e-05 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599796+0000", 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3226000000000003e-05 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64867 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3956736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.491 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.8918252099999999, 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036484356000000003, 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64867 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563456+0000", 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563458+0000", 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9980000000000002e-06 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563463+0000", 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563466+0000", 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599810+0000", 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036343683000000002 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599864+0000", 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3565e-05 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599939+0000", 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5889999999999993e-05 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64868 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3964928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.8918100519999999, 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036573319999999999, 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64868 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563471+0000", 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563473+0000", 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563479+0000", 2024-09-16T13:18:15.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.178e-06 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563483+0000", 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6339999999999999e-06 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599963+0000", 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036480090999999999 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600001+0000", 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8340999999999998e-05 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600044+0000", 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64869 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3973120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:15.500 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.891794078, 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036707818000000003, 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64869 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563487+0000", 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9100000000000003e-07 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563489+0000", 2024-09-16T13:18:15.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563494+0000", 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563498+0000", 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.174e-06 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600061+0000", 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036563127000000001 2024-09-16T13:18:15.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600119+0000", 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8476000000000001e-05 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600194+0000", 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4869000000000004e-05 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64870 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3981312~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.891779036, 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036852528000000002, 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64870 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563502+0000", 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563504+0000", 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9479999999999998e-06 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563509+0000", 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563512+0000", 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3239999999999999e-06 2024-09-16T13:18:15.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600218+0000", 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036705812999999997 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600280+0000", 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2298000000000006e-05 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600354+0000", 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3343000000000006e-05 2024-09-16T13:18:15.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64871 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3989504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.89176433, 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036938311000000001, 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64871 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563518+0000", 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9130000000000001e-06 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563523+0000", 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563529+0000", 2024-09-16T13:18:15.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0710000000000002e-06 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036847958 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600411+0000", 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4388999999999999e-05 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600454+0000", 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3411000000000002e-05 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64872 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3997696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.891747823, 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037077038999999999, 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.513 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64872 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563533+0000", 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563535+0000", 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1270000000000001e-06 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563540+0000", 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3120000000000001e-06 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563544+0000", 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4699999999999998e-06 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600476+0000", 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036931691000000003 2024-09-16T13:18:15.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600543+0000", 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7475e-05 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600610+0000", 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6397999999999997e-05 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64873 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4005888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.891732798, 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037239282999999998, 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64873 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:15.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563550+0000", 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563555+0000", 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563558+0000", 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2009999999999999e-06 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600630+0000", 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037071933000000001 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600702+0000", 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1245e-05 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600787+0000", 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5290000000000005e-05 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64874 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4014080~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.891718352, 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037394846000000002, 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64874 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:15.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0400000000000002e-07 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563565+0000", 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1900000000000002e-06 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563570+0000", 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4199999999999998e-06 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563574+0000", 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600812+0000", 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037237600000000003 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4690000000000006e-05 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600957+0000", 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0463999999999994e-05 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64875 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4022272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.526 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.8917025569999999, 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537883000000001, 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64875 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.65e-07 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563580+0000", 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563587+0000", 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2589999999999997e-06 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563591+0000", 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600982+0000", 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037390117 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601044+0000", 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2788999999999998e-05 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601116+0000", 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1282e-05 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64876 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4030464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.8916852319999999, 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037682571999999998, 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64876 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563596+0000", 2024-09-16T13:18:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563613+0000", 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7935999999999999e-05 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563620+0000", 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4740000000000002e-06 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563624+0000", 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601141+0000", 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037517170000000002 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601203+0000", 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2187999999999995e-05 2024-09-16T13:18:15.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601278+0000", 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64877 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4038656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.891651623, 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037813404000000002, 2024-09-16T13:18:15.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64877 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563631+0000", 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1330000000000002e-06 2024-09-16T13:18:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563637+0000", 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563640+0000", 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601304+0000", 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037664438000000001 2024-09-16T13:18:15.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601372+0000", 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7618000000000002e-05 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601442+0000", 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-05 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.539 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64878 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4046848~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.8916224499999998, 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037956456, 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64878 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563659+0000", 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563661+0000", 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-06 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563667+0000", 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3470000000000004e-06 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563671+0000", 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7170000000000002e-06 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601473+0000", 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037801955999999998 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601537+0000", 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3960000000000004e-05 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601614+0000", 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64879 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4055040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.891605811, 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038485017000000003, 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64879 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:15.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4099999999999998e-07 2024-09-16T13:18:15.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563677+0000", 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1299999999999999e-06 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563682+0000", 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563688+0000", 2024-09-16T13:18:15.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3329999999999996e-06 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601640+0000", 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037951864000000002 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601705+0000", 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5573000000000004e-05 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.602160+0000", 2024-09-16T13:18:15.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000454411 2024-09-16T13:18:15.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.548 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:15.548 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:15.548 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:15.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:15.550 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:15.550 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:15.550 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone31 from 0xddcecf6 (overlap 0x1ab902) to 0xab067 2024-09-16T13:18:15.551 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:15.551 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:15.551 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:15.542+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:15.552 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:15.579 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:15.582 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:15.650 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:15.642+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:15.662 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:15.662 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:15.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:15.750 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:15.744+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:15.776 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:15.776 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:15.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:15.819 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #30, image image_client.0-clone30 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent31 2024-09-16T13:18:15.823 INFO:teuthology.orchestra.run.smithi098.stdout:4603 clone 33 order 21 su 32768 sc 5 2024-09-16T13:18:15.852 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:15.848+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:15.883 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:15.884 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:15.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:15.953 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:15.946+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:16.037 INFO:teuthology.orchestra.run.smithi151.stderr:Exporting 3.15 info 3.15( v 80'162 (0'0,80'162] local-lis/les=23/24 n=29 ec=23/23 lis/c=23/23 les/c/f=24/24/0 sis=23) 2024-09-16T13:18:16.037 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:16.038 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:16.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:16.050 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:a8008361:::rbd_data.112de4fe95f7.00000000000000a9:a# 2024-09-16T13:18:16.050 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:a8008361:::rbd_data.112de4fe95f7.00000000000000a9:head# 2024-09-16T13:18:16.050 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:a8246e5d:::rbd_data.112de4fe95f7.000000000000002f:head# 2024-09-16T13:18:16.050 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:a88a7d7c:::rbd_data.112ddef65b5a.0000000000000026:b# 2024-09-16T13:18:16.050 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:a88a7d7c:::rbd_data.112ddef65b5a.0000000000000026:head# 2024-09-16T13:18:16.051 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:a88e3a25:::rbd_data.112d7dbb182b.000000000000000d:head# 2024-09-16T13:18:16.051 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aa21e89a:::rbd_data.112de4fe95f7.0000000000000083:a# 2024-09-16T13:18:16.052 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aa21e89a:::rbd_data.112de4fe95f7.0000000000000083:head# 2024-09-16T13:18:16.052 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aa2d3911:::rbd_data.112de3ebec3a.000000000000004c:head# 2024-09-16T13:18:16.052 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aa4fd0df:::rbd_data.112d7dbb182b.00000000000001d5:head# 2024-09-16T13:18:16.052 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aa8bad7a:::rbd_data.112de4fe95f7.000000000000004c:a# 2024-09-16T13:18:16.053 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aa8bad7a:::rbd_data.112de4fe95f7.000000000000004c:head# 2024-09-16T13:18:16.053 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aaac9709:::rbd_data.112de4fe95f7.000000000000001f:a# 2024-09-16T13:18:16.054 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:16.048+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:16.054 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aaac9709:::rbd_data.112de4fe95f7.000000000000001f:head# 2024-09-16T13:18:16.054 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aad07d71:::rbd_data.112d8b6bb15.000000000000003e:head# 2024-09-16T13:18:16.054 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:ab32064f:::rbd_data.112d2ce070e7.0000000000000017:head# 2024-09-16T13:18:16.055 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:ab5f8439:::rbd_data.112db432a9c2.0000000000000019:head# 2024-09-16T13:18:16.055 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:ac588cdb:::rbd_data.112dfb79b81.0000000000000006:1a# 2024-09-16T13:18:16.056 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:ac588cdb:::rbd_data.112dfb79b81.0000000000000006:head# 2024-09-16T13:18:16.056 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:ad321648:::rbd_data.112de4fe95f7.0000000000000014:head# 2024-09-16T13:18:16.057 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:head# 2024-09-16T13:18:16.061 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:ae604b58:::rbd_data.112ddef65b5a.000000000000001a:head# 2024-09-16T13:18:16.062 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:ae74f048:::rbd_data.112d420e934f.0000000000000002:head# 2024-09-16T13:18:16.066 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aecdbb92:::rbd_data.112df1cc59bd.0000000000000013:d# 2024-09-16T13:18:16.067 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aecdbb92:::rbd_data.112df1cc59bd.0000000000000013:head# 2024-09-16T13:18:16.068 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aeedf058:::rbd_object_map.112d58339818.000000000000001d:head# 2024-09-16T13:18:16.068 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:aeffa2da:::rbd_data.112de4fe95f7.0000000000000041:head# 2024-09-16T13:18:16.068 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:afdb193a:::rbd_data.112de4fe95f7.00000000000000a7:a# 2024-09-16T13:18:16.068 INFO:teuthology.orchestra.run.smithi151.stderr:Read #3:afdb193a:::rbd_data.112de4fe95f7.00000000000000a7:head# 2024-09-16T13:18:16.068 INFO:teuthology.orchestra.run.smithi151.stderr:Export successful 2024-09-16T13:18:16.151 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:16.151 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:16.153 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:16.148+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:16.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:16.254 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:16.248+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:16.261 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:16.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:16.355 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:16.352+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:16.385 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:16.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:16.456 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:16.452+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:16.495 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:16.495 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:16.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:16.556 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:16.552+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:16.601 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:16.601 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:16.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:16.617 INFO:teuthology.orchestra.run.smithi098.stdout:flattening image image_client.0-clone32 2024-09-16T13:18:16.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 3.15 2024-09-16T13:18:16.657 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:16.652+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:16.716 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:16.716 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:16.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:16.769 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:16.762+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:16.824 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:16.824 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:16.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:16.859 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:16.856+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:16.933 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:16.933 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:16.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:16.959 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:16.956+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:17.045 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:17.045 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:17.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:17.060 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:17.056+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:17.110 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #31, image image_client.0-clone31 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent32 2024-09-16T13:18:17.125 INFO:teuthology.orchestra.run.smithi098.stdout:4608 punch from 0x29b5fe7 to 0x29b7f5c, (0x1f75 bytes) 2024-09-16T13:18:17.134 INFO:teuthology.orchestra.run.smithi098.stdout:4609 write 0x4d0cbac thru 0x4d0e15a (0x15af bytes) 2024-09-16T13:18:17.144 INFO:teuthology.orchestra.run.smithi098.stdout:4610 write 0x9040f6d thru 0x904c586 (0xb61a bytes) 2024-09-16T13:18:17.154 INFO:teuthology.orchestra.run.smithi098.stdout:4612 read 0xad24811 thru 0xad2861a (0x3e0a bytes) 2024-09-16T13:18:17.154 INFO:teuthology.orchestra.run.smithi098.stdout:4613 write 0x35b483c thru 0x35be3e4 (0x9ba9 bytes) 2024-09-16T13:18:17.155 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:17.155 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:17.161 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:17.156+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:17.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:17.165 INFO:teuthology.orchestra.run.smithi098.stdout:4614 writesame 0xa630bbf thru 0xa63562b data_size 0x49(0x4a6d bytes) 2024-09-16T13:18:17.173 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:17.173 INFO:teuthology.orchestra.run.smithi098.stdout:4617 read 0xc36ac61 thru 0xc379bc0 (0xef60 bytes) 2024-09-16T13:18:17.174 INFO:teuthology.orchestra.run.smithi098.stdout:4618 punch from 0xc60cc4 to 0xc6868b, (0x79c7 bytes) 2024-09-16T13:18:17.174 INFO:teuthology.orchestra.run.smithi098.stdout:4619 flatten 2024-09-16T13:18:17.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:18:17.261 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:17.254+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:17.297 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:17.298 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:17.298 INFO:teuthology.orchestra.run.smithi098.stdout:4621 write 0xc19f6e2 thru 0xc1ab517 (0xbe36 bytes) 2024-09-16T13:18:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:4622 write 0x9c50b3a thru 0x9c53078 (0x253f bytes) 2024-09-16T13:18:17.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:17.309 INFO:teuthology.orchestra.run.smithi098.stdout:4623 read 0x6767626 thru 0x6768449 (0xe24 bytes) 2024-09-16T13:18:17.309 INFO:teuthology.orchestra.run.smithi098.stdout:4624 punch from 0xd1c6276 to 0xd1d16e2, (0xb46c bytes) 2024-09-16T13:18:17.310 INFO:teuthology.orchestra.run.smithi098.stdout:4625 write 0x61b2e86 thru 0x61b43a9 (0x1524 bytes) 2024-09-16T13:18:17.314 INFO:teuthology.orchestra.run.smithi098.stdout:4626 write 0xdf0e01 thru 0xdf5873 (0x4a73 bytes) 2024-09-16T13:18:17.330 INFO:teuthology.orchestra.run.smithi098.stdout:4629 write 0xe427eb1 thru 0xe42badc (0x3c2c bytes) 2024-09-16T13:18:17.347 INFO:teuthology.orchestra.run.smithi098.stdout:4630 write 0xed1806c thru 0xed18d2a (0xcbf bytes) 2024-09-16T13:18:17.351 INFO:teuthology.orchestra.run.smithi098.stdout:4631 write 0x5d2ae9f thru 0x5d37d23 (0xce85 bytes) 2024-09-16T13:18:17.362 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:17.354+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:17.364 INFO:teuthology.orchestra.run.smithi098.stdout:4632 read 0x56884db thru 0x5691fa6 (0x9acc bytes) 2024-09-16T13:18:17.364 INFO:teuthology.orchestra.run.smithi098.stdout:4633 write 0xcf9baa4 thru 0xcf9da34 (0x1f91 bytes) 2024-09-16T13:18:17.369 INFO:teuthology.orchestra.run.smithi098.stdout:4635 writesame 0x68ba1b6 thru 0x68c1118 data_size 0x5(0x6f63 bytes) 2024-09-16T13:18:17.382 INFO:teuthology.orchestra.run.smithi098.stdout:4636 trunc from 0xed18d2b to 0xe142743 2024-09-16T13:18:17.400 INFO:teuthology.orchestra.run.smithi098.stdout:4640 read 0x806d771 thru 0x80717d7 (0x4067 bytes) 2024-09-16T13:18:17.406 INFO:teuthology.orchestra.run.smithi098.stdout:4641 trunc from 0xe142743 to 0xa61f8ca 2024-09-16T13:18:17.412 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:17.412 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:17.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:17.426 INFO:teuthology.orchestra.run.smithi098.stdout:4642 punch from 0x8417830 to 0x8423cb8, (0xc488 bytes) 2024-09-16T13:18:17.426 INFO:teuthology.orchestra.run.smithi098.stdout:4643 write 0x178acfe thru 0x17946e9 (0x99ec bytes) 2024-09-16T13:18:17.434 INFO:teuthology.orchestra.run.smithi098.stdout:4644 writesame 0x2723f2d thru 0x2732c0c data_size 0x50(0xece0 bytes) 2024-09-16T13:18:17.442 INFO:teuthology.orchestra.run.smithi098.stdout:4645 writesame 0x244458e thru 0x2452c0f data_size 0x46(0xe682 bytes) 2024-09-16T13:18:17.451 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:17.451 INFO:teuthology.orchestra.run.smithi098.stdout:4648 punch from 0x2b85628 to 0x2b86cde, (0x16b6 bytes) 2024-09-16T13:18:17.458 INFO:teuthology.orchestra.run.smithi098.stdout:4649 trunc from 0xa61f8ca to 0xececcbb 2024-09-16T13:18:17.462 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:17.456+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:17.467 INFO:teuthology.orchestra.run.smithi098.stdout:4650 read 0x293ae06 thru 0x2943033 (0x822e bytes) 2024-09-16T13:18:17.467 INFO:teuthology.orchestra.run.smithi098.stdout:4651 write 0x6f8f045 thru 0x6f949ed (0x59a9 bytes) 2024-09-16T13:18:17.475 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:17.476 INFO:teuthology.orchestra.run.smithi098.stdout:4653 read 0x37ac01d thru 0x37b1f6a (0x5f4e bytes) 2024-09-16T13:18:17.476 INFO:teuthology.orchestra.run.smithi098.stdout:4654 write 0x4240169 thru 0x424cd0a (0xcba2 bytes) 2024-09-16T13:18:17.484 INFO:teuthology.orchestra.run.smithi098.stdout:4655 read 0xb0f47bb thru 0xb0f7915 (0x315b bytes) 2024-09-16T13:18:17.484 INFO:teuthology.orchestra.run.smithi098.stdout:4656 punch from 0xb6cbb96 to 0xb6da7fe, (0xec68 bytes) 2024-09-16T13:18:17.484 INFO:teuthology.orchestra.run.smithi098.stdout:4657 write 0xe194eca thru 0xe19f9fb (0xab32 bytes) 2024-09-16T13:18:17.491 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:17.491 INFO:teuthology.orchestra.run.smithi098.stdout:4659 read 0xbe15d7a thru 0xbe22084 (0xc30b bytes) 2024-09-16T13:18:17.491 INFO:teuthology.orchestra.run.smithi098.stdout:4660 read 0x52ad11e thru 0x52b3041 (0x5f24 bytes) 2024-09-16T13:18:17.491 INFO:teuthology.orchestra.run.smithi098.stdout:4661 read 0xb840ed5 thru 0xb842764 (0x1890 bytes) 2024-09-16T13:18:17.492 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:17.492 INFO:teuthology.orchestra.run.smithi098.stdout:4665 read 0x687c30d thru 0x68827a1 (0x6495 bytes) 2024-09-16T13:18:17.492 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:17.492 INFO:teuthology.orchestra.run.smithi098.stdout:4668 write 0xe522d82 thru 0xe52312b (0x3aa bytes) 2024-09-16T13:18:17.496 INFO:teuthology.orchestra.run.smithi098.stdout:4669 writesame 0x62bd1f thru 0x632704 data_size 0x5(0x69e6 bytes) 2024-09-16T13:18:17.503 INFO:teuthology.orchestra.run.smithi098.stdout:4670 writesame 0x8373cdc thru 0x837958a data_size 0x1(0x58af bytes) 2024-09-16T13:18:17.530 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:17.530 INFO:teuthology.orchestra.run.smithi098.stdout:4673 read 0x809c3af thru 0x809d1e4 (0xe36 bytes) 2024-09-16T13:18:17.530 INFO:teuthology.orchestra.run.smithi098.stdout:4674 read 0x8480136 thru 0x8484448 (0x4313 bytes) 2024-09-16T13:18:17.530 INFO:teuthology.orchestra.run.smithi098.stdout:4675 writesame 0x93e7184 thru 0x93f6ef3 data_size 0x28(0xfd70 bytes) 2024-09-16T13:18:17.537 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:17.537 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:17.538 INFO:teuthology.orchestra.run.smithi098.stdout:4676 punch from 0xc4c0ebf to 0xc4cda2e, (0xcb6f bytes) 2024-09-16T13:18:17.538 INFO:teuthology.orchestra.run.smithi098.stdout:4678 read 0x801ba28 thru 0x801fe3c (0x4415 bytes) 2024-09-16T13:18:17.539 INFO:teuthology.orchestra.run.smithi098.stdout:4679 writesame 0xaa23854 thru 0xaa33366 data_size 0x19(0xfb13 bytes) 2024-09-16T13:18:17.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:17.548 INFO:teuthology.orchestra.run.smithi098.stdout:4681 writesame 0x630a569 thru 0x63110e8 data_size 0x40(0x6b80 bytes) 2024-09-16T13:18:17.569 INFO:teuthology.orchestra.run.smithi098.stdout:4682 trunc from 0xececcbb to 0x55906ac 2024-09-16T13:18:17.569 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:17.564+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:17.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:17.593 INFO:teuthology.orchestra.run.smithi098.stdout:4683 punch from 0x1b84932 to 0x1b91aa3, (0xd171 bytes) 2024-09-16T13:18:17.593 INFO:teuthology.orchestra.run.smithi098.stdout:4685 trunc from 0x55906ac to 0x5806761 2024-09-16T13:18:17.610 INFO:teuthology.orchestra.run.smithi098.stdout:4686 write 0x617fcd2 thru 0x618c2c5 (0xc5f4 bytes) 2024-09-16T13:18:17.618 INFO:teuthology.orchestra.run.smithi098.stdout:4687 punch from 0x5ed548d to 0x5ed552d, (0xa0 bytes) 2024-09-16T13:18:17.618 INFO:teuthology.orchestra.run.smithi098.stdout:4688 read 0x203a14f thru 0x2048c11 (0xeac3 bytes) 2024-09-16T13:18:17.619 INFO:teuthology.orchestra.run.smithi098.stdout:4689 write 0x1d03b93 thru 0x1d111f5 (0xd663 bytes) 2024-09-16T13:18:17.624 INFO:teuthology.orchestra.run.smithi098.stdout:4690 punch from 0x115da21 to 0x11646bb, (0x6c9a bytes) 2024-09-16T13:18:17.624 INFO:teuthology.orchestra.run.smithi098.stdout:4691 writesame 0x3dbaa84 thru 0x3dbe064 data_size 0x1(0x35e1 bytes) 2024-09-16T13:18:17.637 INFO:teuthology.orchestra.run.smithi098.stdout:4692 read 0xe0ae1b thru 0xe0d0ab (0x2291 bytes) 2024-09-16T13:18:17.637 INFO:teuthology.orchestra.run.smithi098.stdout:4694 write 0x583323e thru 0x5836fdd (0x3da0 bytes) 2024-09-16T13:18:17.649 INFO:teuthology.orchestra.run.smithi098.stdout:4695 trunc from 0x618c2c6 to 0x9d9f490 2024-09-16T13:18:17.658 INFO:teuthology.orchestra.run.smithi098.stdout:4696 read 0x79a5001 thru 0x79ada86 (0x8a86 bytes) 2024-09-16T13:18:17.658 INFO:teuthology.orchestra.run.smithi098.stdout:4698 read 0x85570f1 thru 0x855b756 (0x4666 bytes) 2024-09-16T13:18:17.658 INFO:teuthology.orchestra.run.smithi098.stdout:4699 write 0x4f206ae thru 0x4f22dfd (0x2750 bytes) 2024-09-16T13:18:17.663 INFO:teuthology.orchestra.run.smithi098.stdout:4700 writesame 0x1430184 thru 0x1433c83 data_size 0x20(0x3b00 bytes) 2024-09-16T13:18:17.664 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:17.660+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:17.668 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:17.668 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:17.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:17.687 INFO:teuthology.orchestra.run.smithi098.stdout:4701 writesame 0xea27d85 thru 0xea34cf2 data_size 0x7(0xcf6e bytes) 2024-09-16T13:18:17.688 INFO:teuthology.orchestra.run.smithi098.stdout:4702 write 0x8554ba8 thru 0x85641e8 (0xf641 bytes) 2024-09-16T13:18:17.695 INFO:teuthology.orchestra.run.smithi098.stdout:4703 read 0xc23bd20 thru 0xc24897d (0xcc5e bytes) 2024-09-16T13:18:17.696 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:17.696 INFO:teuthology.orchestra.run.smithi098.stdout:4706 punch from 0xdbe2d2d to 0xdbe41a7, (0x147a bytes) 2024-09-16T13:18:17.696 INFO:teuthology.orchestra.run.smithi098.stdout:4709 read 0x5a2627d thru 0x5a3386c (0xd5f0 bytes) 2024-09-16T13:18:17.698 INFO:teuthology.orchestra.run.smithi098.stdout:4710 punch from 0xcfee431 to 0xcffc04c, (0xdc1b bytes) 2024-09-16T13:18:17.699 INFO:teuthology.orchestra.run.smithi098.stdout:4711 trunc from 0xea34cf3 to 0xdd00eeb 2024-09-16T13:18:17.707 INFO:teuthology.orchestra.run.smithi098.stdout:4712 write 0x7e860ab thru 0x7e8967b (0x35d1 bytes) 2024-09-16T13:18:17.714 INFO:teuthology.orchestra.run.smithi098.stdout:4713 read 0x970e0d2 thru 0x97196ea (0xb619 bytes) 2024-09-16T13:18:17.715 INFO:teuthology.orchestra.run.smithi098.stdout:4714 writesame 0x51120ea thru 0x511a674 data_size 0x1(0x858b bytes) 2024-09-16T13:18:17.721 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:17.739 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:17.739 INFO:teuthology.orchestra.run.smithi098.stdout:4717 punch from 0x7a82b53 to 0x7a874e1, (0x498e bytes) 2024-09-16T13:18:17.739 INFO:teuthology.orchestra.run.smithi098.stdout:4718 write 0x94f0d5c thru 0x94ff2fd (0xe5a2 bytes) 2024-09-16T13:18:17.745 INFO:teuthology.orchestra.run.smithi098.stdout:4719 read 0x896e3e8 thru 0x8972c0a (0x4823 bytes) 2024-09-16T13:18:17.746 INFO:teuthology.orchestra.run.smithi098.stdout:4720 read 0x5ee5e42 thru 0x5ee6d20 (0xedf bytes) 2024-09-16T13:18:17.746 INFO:teuthology.orchestra.run.smithi098.stdout:4721 punch from 0x7f49064 to 0x7f5801d, (0xefb9 bytes) 2024-09-16T13:18:17.747 INFO:teuthology.orchestra.run.smithi098.stdout:4722 writesame 0x854d35a thru 0x854faf7 data_size 0xb(0x279e bytes) 2024-09-16T13:18:17.749 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:17.752 INFO:teuthology.orchestra.run.smithi098.stdout:4723 clone 34 order 25 su 131072 sc 12 2024-09-16T13:18:17.764 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:17.760+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:17.771 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:17.777 INFO:teuthology.orchestra.run.smithi151.stdout: marking collection for removal 2024-09-16T13:18:17.777 INFO:teuthology.orchestra.run.smithi151.stdout:setting '_remove' omap key 2024-09-16T13:18:17.782 INFO:teuthology.orchestra.run.smithi151.stdout:finish_remove_pgs 3.15_head removing 3.15 2024-09-16T13:18:17.783 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:17.783 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:17.783 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:17.790 INFO:teuthology.orchestra.run.smithi151.stdout:Remove successful 2024-09-16T13:18:17.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:17.794 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:17.813 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:17.833 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:17.860 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:17.861 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:17.861 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:17.861 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:17.861 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:17.861 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:17.862 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:17.862 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:17.862 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:17.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:17.866 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:17.858+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:17.885 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:17.886 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:17.886 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:17.886 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:17.886 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:17.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:17.966 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:17.958+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:18.044 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:18.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64860 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3899392~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4757067939999997, 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035719773000000003, 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64860 2024-09-16T13:18:18.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563336+0000", 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563342+0000", 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3240000000000002e-06 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563345+0000", 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.598973+0000", 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035627631999999999 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599006+0000", 2024-09-16T13:18:18.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3495999999999999e-05 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599054+0000", 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7460999999999998e-05 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64861 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3907584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.049 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4756919130000004, 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035815461, 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64861 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:18:18.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563351+0000", 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.854e-06 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563356+0000", 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5940000000000001e-06 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563360+0000", 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4060000000000001e-06 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599076+0000", 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035716171999999997 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599116+0000", 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9777999999999998e-05 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.053 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599164+0000", 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8307999999999999e-05 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64862 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.47567732, 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035954097999999997, 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.054 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64862 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:18.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563364+0000", 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563365+0000", 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.821e-06 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563371+0000", 2024-09-16T13:18:18.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0769999999999998e-06 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563374+0000", 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3560000000000001e-06 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599187+0000", 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035813153 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599252+0000", 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4873e-05 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599317+0000", 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5438000000000006e-05 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64863 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3923968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4756517660000004, 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036051842000000001, 2024-09-16T13:18:18.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64863 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563392+0000", 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3140000000000002e-06 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563398+0000", 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.066e-06 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563402+0000", 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599339+0000", 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035937678000000001 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599377+0000", 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599441+0000", 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3380999999999995e-05 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64864 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3932160~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4756353620000002, 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036187015000000003, 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64864 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563406+0000", 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563411+0000", 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563417+0000", 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563421+0000", 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599458+0000", 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036036870999999998 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599515+0000", 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7067000000000001e-05 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599592+0000", 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7457000000000001e-05 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.066 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64865 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3940352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4756159719999999, 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036265720000000001, 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64865 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:18:18.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563427+0000", 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.001e-06 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563433+0000", 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5489999999999997e-06 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563436+0000", 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.484e-06 2024-09-16T13:18:18.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599613+0000", 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036176815000000001 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599646+0000", 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3192999999999999e-05 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599690+0000", 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4240000000000003e-05 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64866 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3948544~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4756001940000001, 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036355749999999999, 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64866 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563441+0000", 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:18.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563443+0000", 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0870000000000001e-06 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563448+0000", 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.062e-06 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563451+0000", 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599712+0000", 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036261007999999997 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599753+0000", 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0476999999999999e-05 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599796+0000", 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3226000000000003e-05 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64867 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3956736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.475585454, 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036484356000000003, 2024-09-16T13:18:18.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64867 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563456+0000", 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563458+0000", 2024-09-16T13:18:18.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9980000000000002e-06 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563463+0000", 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563466+0000", 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599810+0000", 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036343683000000002 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599864+0000", 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3565e-05 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599939+0000", 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5889999999999993e-05 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.079 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64868 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3964928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4755702959999999, 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036573319999999999, 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.080 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64868 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:18.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563471+0000", 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563473+0000", 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563479+0000", 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.178e-06 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563483+0000", 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6339999999999999e-06 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599963+0000", 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036480090999999999 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600001+0000", 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8340999999999998e-05 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600044+0000", 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64869 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3973120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4755543219999998, 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036707818000000003, 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64869 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563487+0000", 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9100000000000003e-07 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563489+0000", 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563494+0000", 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563498+0000", 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.174e-06 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600061+0000", 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036563127000000001 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600119+0000", 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8476000000000001e-05 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600194+0000", 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4869000000000004e-05 2024-09-16T13:18:18.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64870 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3981312~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4755392799999996, 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036852528000000002, 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.089 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64870 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563502+0000", 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563504+0000", 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9479999999999998e-06 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563509+0000", 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:18:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563512+0000", 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3239999999999999e-06 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600218+0000", 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036705812999999997 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600280+0000", 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2298000000000006e-05 2024-09-16T13:18:18.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600354+0000", 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3343000000000006e-05 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64871 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3989504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4755245739999996, 2024-09-16T13:18:18.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036938311000000001, 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64871 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:18.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563518+0000", 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9130000000000001e-06 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563523+0000", 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563529+0000", 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0710000000000002e-06 2024-09-16T13:18:18.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036847958 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600411+0000", 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4388999999999999e-05 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600454+0000", 2024-09-16T13:18:18.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3411000000000002e-05 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64872 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3997696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4755080669999998, 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037077038999999999, 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.098 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64872 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563533+0000", 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563535+0000", 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1270000000000001e-06 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563540+0000", 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3120000000000001e-06 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563544+0000", 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4699999999999998e-06 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600476+0000", 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036931691000000003 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600543+0000", 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7475e-05 2024-09-16T13:18:18.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600610+0000", 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6397999999999997e-05 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64873 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4005888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4754930420000001, 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037239282999999998, 2024-09-16T13:18:18.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64873 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563550+0000", 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563555+0000", 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563558+0000", 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2009999999999999e-06 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600630+0000", 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037071933000000001 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600702+0000", 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1245e-05 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600787+0000", 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5290000000000005e-05 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.106 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64874 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4014080~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4754785960000003, 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037394846000000002, 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64874 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0400000000000002e-07 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563565+0000", 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1900000000000002e-06 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563570+0000", 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4199999999999998e-06 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563574+0000", 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600812+0000", 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037237600000000003 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4690000000000006e-05 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600957+0000", 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0463999999999994e-05 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.110 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64875 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4022272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4754628009999999, 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537883000000001, 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64875 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.111 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:18.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.65e-07 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563580+0000", 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563587+0000", 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2589999999999997e-06 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563591+0000", 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600982+0000", 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037390117 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601044+0000", 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2788999999999998e-05 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601116+0000", 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1282e-05 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64876 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4030464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.475445476, 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037682571999999998, 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64876 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563596+0000", 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563613+0000", 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7935999999999999e-05 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563620+0000", 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4740000000000002e-06 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563624+0000", 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601141+0000", 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037517170000000002 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601203+0000", 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2187999999999995e-05 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601278+0000", 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64877 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4038656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.475411867, 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037813404000000002, 2024-09-16T13:18:18.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64877 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563631+0000", 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1330000000000002e-06 2024-09-16T13:18:18.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563637+0000", 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563640+0000", 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601304+0000", 2024-09-16T13:18:18.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037664438000000001 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601372+0000", 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7618000000000002e-05 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601442+0000", 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-05 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64878 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4046848~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4753826940000003, 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037956456, 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.124 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64878 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563659+0000", 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563661+0000", 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-06 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563667+0000", 2024-09-16T13:18:18.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3470000000000004e-06 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563671+0000", 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7170000000000002e-06 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601473+0000", 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037801955999999998 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601537+0000", 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3960000000000004e-05 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601614+0000", 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64879 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4055040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.128 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4753660550000003, 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038485017000000003, 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64879 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4099999999999998e-07 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563677+0000", 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1299999999999999e-06 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563682+0000", 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563688+0000", 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3329999999999996e-06 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601640+0000", 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037951864000000002 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601705+0000", 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5573000000000004e-05 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.602160+0000", 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000454411 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.133 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:18.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:18.135 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:18.128+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:18.234 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:18:18.234 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:18.228+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:18.246 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:18.246 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:18.246 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:18.246 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:18.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:18.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.15.7 2024-09-16T13:18:18.335 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:18.332+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:18.346 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:18.346 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:18.346 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:18.347 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:18.347 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:18.356 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:18.436 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:18.432+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 74.446180322999993, 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:18.463 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 74.445394824000005, 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:18:18.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:18:18.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 74.444227901999994, 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2447 3.9 e31/23 3:93287c75:::rbd_data.112dbfd5d823.00000000000001c0:head v 31'19, mlcod=31'17)", 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 74.441353270999997, 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028950540000000002, 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012220+0000", 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3399999999999999e-07 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012226+0000", 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.434e-06 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012238+0000", 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2626000000000001e-05 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012243+0000", 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9459999999999997e-06 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012296+0000", 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2975e-05 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.012335+0000", 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8488999999999997e-05 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015067+0000", 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027326579999999998 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015115+0000", 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7292000000000001e-05 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 74.438339530999997, 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:18.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:18:18.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2551 3.4 e31/23 3:203672b7:::rbd_data.112dbfd5d823.0000000000000018:head v 31'6, mlcod=27'4)", 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:18:18.480 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 74.413135327999996, 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0030192109999999999, 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040437+0000", 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040439+0000", 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2419999999999999e-06 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040611+0000", 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000172001 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040634+0000", 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3308000000000001e-05 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040642+0000", 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6650000000000003e-06 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040727+0000", 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5345000000000004e-05 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.040771+0000", 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4456999999999998e-05 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043396+0000", 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0026244620000000001 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.043457+0000", 2024-09-16T13:18:18.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0730999999999998e-05 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 73.646986439000003, 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:18:18.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:5115 3.4 e34/23 3:20725922:::rbd_data.112d64676e7f.00000000000002f4:head v 34'14, mlcod=31'12)", 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 70.661938332999995, 2024-09-16T13:18:18.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029400680000000001, 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791634+0000", 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791635+0000", 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4200000000000005e-07 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791643+0000", 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.157e-06 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791660+0000", 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6246e-05 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791666+0000", 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4169999999999997e-06 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791743+0000", 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7000000000000001e-05 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.791756+0000", 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2729e-05 2024-09-16T13:18:18.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794523+0000", 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0027675949999999999 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:07.794575+0000", 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1082000000000003e-05 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 66.394884136000002, 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:18:18.497 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 66.394274695999997, 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:18:18.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:18.502 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.896990881999997, 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.895890066000007, 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.895750426000006, 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.511 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:18:18.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:18.515 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.895249895999996, 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:18:18.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 60.900080510000002, 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:18.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 50.643666668999998, 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.527 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:18:18.528 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 50.031103561999998, 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:18.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:18:18.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:18:18.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:22854 3.9 e57/23 3:924a2914:::rbd_data.112db5030c67.0000000000000008:head v 57'198, mlcod=57'196)", 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.912582997999998, 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029464980000000001, 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540990+0000", 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.540995+0000", 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3469999999999998e-06 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541018+0000", 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2382999999999999e-05 2024-09-16T13:18:18.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541079+0000", 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1773999999999995e-05 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.541094+0000", 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4414e-05 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.542159+0000", 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010653749999999999 2024-09-16T13:18:18.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.542208+0000", 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8517000000000003e-05 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.543797+0000", 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0015893210000000001 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:41.543936+0000", 2024-09-16T13:18:18.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000139367 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.192997391999999, 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 22.828870059, 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:18:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.545 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:18.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:18.547 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #32, image image_client.0-clone32 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent33 2024-09-16T13:18:18.548 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:18.538+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:18.648 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:18.638+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:18.699 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:18.699 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:18.699 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:18.699 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:18.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:18.749 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:18.744+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:18.809 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:18.809 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:18.809 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:18.809 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:18.809 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:18.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:18.863 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:18.860+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:18.925 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:18.925 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:18.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:18.925 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:18.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.925 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62278 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.925 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6031489429999999, 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027907498999999999, 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62278 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:18.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:18.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315829+0000", 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.79e-06 2024-09-16T13:18:18.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315835+0000", 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.682e-06 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315838+0000", 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0730000000000001e-06 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343640+0000", 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027802384999999999 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343683+0000", 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3058e-05 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343734+0000", 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1104999999999999e-05 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6031343820000004, 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:18:18.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:18.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:18:18.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.936 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6031104159999998, 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:18:18.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6030952459999996, 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:18:18.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.603080823, 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:18.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:18.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:18:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:18:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:18:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:18:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6030595779999999, 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.953 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:18:18.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.954 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:18.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:18:18.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6030450089999997, 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:18:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:18:18.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.963 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6030262400000002, 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.964 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:18.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:18.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:18:18.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6030113449999996, 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:18:18.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:18:18.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6029971339999998, 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:18:18.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6029837650000003, 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:18:18.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.981 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:18.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:18.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:18:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:18:18.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:18:18.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:18:18.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:18:18.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:18:18.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:18:18.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:18:18.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:18:18.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.986 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.986 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:18.986 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6029690829999996, 2024-09-16T13:18:18.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:18:18.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.986 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.986 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.986 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.986 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.987 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:18:18.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.987 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:18.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:18.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:18.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:18:18.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:18:18.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:18.991 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6029539140000004, 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:18:18.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:18:18.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.996 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6029392739999997, 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:18.997 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:18.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:18:18.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:18:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:18:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:18:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:18:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6029085460000001, 2024-09-16T13:18:19.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:19.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:19.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:19.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.007 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6028875290000002, 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:19.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:19.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:18:19.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:19.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.602867421, 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:19.013 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:19.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:18:19.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:18:19.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6028475179999999, 2024-09-16T13:18:19.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:18:19.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:19.019 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:19.019 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:19.019 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:19.019 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.019 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:18:19.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.019 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:19.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:18:19.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:18:19.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6028323049999997, 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.024 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:19.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:18:19.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.028 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 5.6028167709999996, 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:19.029 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:19.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:18:19.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:18:19.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.034 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:19.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:19.034 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:19.034 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:19.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:19.036 INFO:teuthology.orchestra.run.smithi098.stdout:4724 writesame 0x2263be thru 0x22bf63 data_size 0x1(0x5ba6 bytes) 2024-09-16T13:18:19.036 INFO:teuthology.orchestra.run.smithi098.stdout:4725 clone 35 order 21 su 16384 sc 6 2024-09-16T13:18:19.037 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:19.032+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:19.136 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:19.126+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:19.143 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:19.143 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:19.143 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:19.143 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:19.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:19.237 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:19.232+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:19.259 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:19.259 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:19.259 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:19.259 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:19.259 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:19.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:19.338 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:19.332+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:19.365 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:19.365 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:19.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.1501575830000004, 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:18:19.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:18:19.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:18:19.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64676 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6717440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.558704+0000", 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8012373999999998, 2024-09-16T13:18:19.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038895784000000003, 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64676 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558704+0000", 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558704+0000", 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558704+0000", 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000002e-07 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558706+0000", 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1160000000000002e-06 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558712+0000", 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4140000000000002e-06 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558715+0000", 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4539999999999999e-06 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.594853+0000", 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.036138116999999997 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.594930+0000", 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6243999999999997e-05 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597600+0000", 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026700539999999998 2024-09-16T13:18:19.375 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64686 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6799360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.558938+0000", 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8010036439999997, 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038103766999999997, 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.376 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64686 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558938+0000", 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558938+0000", 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558938+0000", 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558940+0000", 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.136e-06 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558945+0000", 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3399999999999997e-06 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558949+0000", 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5839999999999999e-06 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.596876+0000", 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.03792645 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.596944+0000", 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8967999999999994e-05 2024-09-16T13:18:19.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597041+0000", 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6922999999999993e-05 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64687 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6807552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.558953+0000", 2024-09-16T13:18:19.380 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8009886440000002, 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038272869000000001, 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64687 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558953+0000", 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558953+0000", 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558953+0000", 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9099999999999999e-07 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558955+0000", 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9910000000000001e-06 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558960+0000", 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3140000000000003e-06 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558967+0000", 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1770000000000003e-06 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597065+0000", 2024-09-16T13:18:19.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038098672 2024-09-16T13:18:19.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597132+0000", 2024-09-16T13:18:19.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6657999999999998e-05 2024-09-16T13:18:19.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597226+0000", 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3665999999999994e-05 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64688 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6815744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.558971+0000", 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8009707630000005, 2024-09-16T13:18:19.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038439464, 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64688 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558971+0000", 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558971+0000", 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558971+0000", 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5799999999999999e-07 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558973+0000", 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0729999999999999e-06 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558979+0000", 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4229999999999996e-06 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558982+0000", 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:18:19.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597246+0000", 2024-09-16T13:18:19.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038264063000000001 2024-09-16T13:18:19.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597312+0000", 2024-09-16T13:18:19.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5974000000000002e-05 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597410+0000", 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7684999999999997e-05 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64689 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6823936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.558987+0000", 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8009541169999999, 2024-09-16T13:18:19.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038595690000000002, 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64689 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558987+0000", 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558987+0000", 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558988+0000", 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558990+0000", 2024-09-16T13:18:19.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.182e-06 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.558995+0000", 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2639999999999999e-06 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559000+0000", 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8210000000000001e-06 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597431+0000", 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038431532999999997 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597494+0000", 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2881999999999999e-05 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597583+0000", 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8653000000000007e-05 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.394 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64690 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6832128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559004+0000", 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8009375030000001, 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038788806000000002, 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64690 2024-09-16T13:18:19.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559004+0000", 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559004+0000", 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559004+0000", 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559006+0000", 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.962e-06 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559012+0000", 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3990000000000003e-06 2024-09-16T13:18:19.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559016+0000", 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3850000000000004e-06 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597629+0000", 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038612550000000002 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597696+0000", 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7965000000000004e-05 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597793+0000", 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6099000000000003e-05 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64691 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6840320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559019+0000", 2024-09-16T13:18:19.399 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8009218210000002, 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040968721, 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64691 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559019+0000", 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559019+0000", 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559020+0000", 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.53e-07 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559022+0000", 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0999999999999998e-06 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559029+0000", 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.878e-06 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559036+0000", 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000003e-06 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.597820+0000", 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038783718000000002 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.599886+0000", 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020662549999999999 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.599988+0000", 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101927 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64692 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6848512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559054+0000", 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8008869829999998, 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041108762, 2024-09-16T13:18:19.404 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64692 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559054+0000", 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559054+0000", 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559055+0000", 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5400000000000001e-07 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559057+0000", 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3769999999999999e-06 2024-09-16T13:18:19.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559063+0000", 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559067+0000", 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7230000000000002e-06 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600011+0000", 2024-09-16T13:18:19.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040944095999999999 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600108+0000", 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6917999999999996e-05 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600163+0000", 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5012000000000001e-05 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64693 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6856704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559071+0000", 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8008700940000004, 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041219474999999998, 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64693 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559071+0000", 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559071+0000", 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559072+0000", 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9900000000000001e-07 2024-09-16T13:18:19.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559078+0000", 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1539999999999999e-06 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559084+0000", 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7860000000000002e-06 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559087+0000", 2024-09-16T13:18:19.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6459999999999999e-06 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600176+0000", 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041088434 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600213+0000", 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7413000000000002e-05 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600291+0000", 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7542999999999999e-05 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64694 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6864896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559093+0000", 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8008486530000001, 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041365622999999997, 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.413 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64694 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559093+0000", 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559093+0000", 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559093+0000", 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2899999999999999e-07 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559095+0000", 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0980000000000001e-06 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559101+0000", 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4240000000000001e-06 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559105+0000", 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4989999999999996e-06 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600311+0000", 2024-09-16T13:18:19.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041206185999999999 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5061999999999995e-05 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600458+0000", 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1924999999999999e-05 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.417 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64695 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6873088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559110+0000", 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8008309579999997, 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041445530000000001, 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64695 2024-09-16T13:18:19.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559110+0000", 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559110+0000", 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559111+0000", 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9799999999999999e-07 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559113+0000", 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559460+0000", 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000347022 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559469+0000", 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8699999999999998e-06 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600471+0000", 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041002200000000003 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600506+0000", 2024-09-16T13:18:19.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5426999999999998e-05 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600556+0000", 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9669999999999997e-05 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64696 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6881280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559476+0000", 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8004653499999996, 2024-09-16T13:18:19.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041184922999999998, 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64696 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559476+0000", 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559476+0000", 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559477+0000", 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4700000000000001e-07 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559481+0000", 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0389999999999998e-06 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559489+0000", 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2169999999999994e-06 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559493+0000", 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9770000000000002e-06 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600568+0000", 2024-09-16T13:18:19.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041075118000000001 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600606+0000", 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7793999999999998e-05 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600661+0000", 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5031000000000002e-05 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.426 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64697 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6889472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.559498+0000", 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8004433039999999, 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041268907000000001, 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64697 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559498+0000", 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.559498+0000", 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562851+0000", 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033532179999999998 2024-09-16T13:18:19.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562858+0000", 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7440000000000001e-06 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562871+0000", 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3251000000000001e-05 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562878+0000", 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.832e-06 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600673+0000", 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037795424000000001 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600709+0000", 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5457000000000003e-05 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600767+0000", 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7980999999999999e-05 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64698 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6897664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562885+0000", 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.7970562149999996, 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037978407999999998, 2024-09-16T13:18:19.431 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64698 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562885+0000", 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562885+0000", 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562886+0000", 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5899999999999995e-07 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562889+0000", 2024-09-16T13:18:19.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2459999999999999e-06 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562897+0000", 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3129999999999999e-06 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562902+0000", 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1359999999999999e-06 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600780+0000", 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037878214 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600815+0000", 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5512999999999997e-05 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600863+0000", 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8226999999999998e-05 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.435 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64699 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6905856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562907+0000", 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.7970342029999999, 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038061630999999999, 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.436 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64699 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562907+0000", 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562907+0000", 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562908+0000", 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562910+0000", 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6570000000000001e-06 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562916+0000", 2024-09-16T13:18:19.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2299999999999996e-06 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562920+0000", 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037956075999999998 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600918+0000", 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1332999999999999e-05 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600969+0000", 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0976000000000001e-05 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.440 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64700 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6914048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562924+0000", 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.7970172160000004, 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038145167000000001, 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64700 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562924+0000", 2024-09-16T13:18:19.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562924+0000", 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562925+0000", 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562927+0000", 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1629999999999999e-06 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562947+0000", 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9930000000000001e-05 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562951+0000", 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.403e-06 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.600981+0000", 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038029962 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601018+0000", 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7503000000000003e-05 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601069+0000", 2024-09-16T13:18:19.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0764999999999999e-05 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64701 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6922240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562956+0000", 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.7969854170000001, 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038235064999999999, 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64701 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562956+0000", 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562956+0000", 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562957+0000", 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562959+0000", 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5330000000000001e-06 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562965+0000", 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2349999999999996e-06 2024-09-16T13:18:19.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562969+0000", 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8739999999999997e-06 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601082+0000", 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038112372999999998 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601124+0000", 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2743000000000001e-05 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601191+0000", 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6612000000000005e-05 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64702 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6930432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562973+0000", 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.796968219, 2024-09-16T13:18:19.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038316650000000001, 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64702 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562973+0000", 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562973+0000", 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562974+0000", 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9800000000000004e-07 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562976+0000", 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1569999999999998e-06 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562981+0000", 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562985+0000", 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601204+0000", 2024-09-16T13:18:19.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038218528000000002 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601240+0000", 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6000000000000001e-05 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601290+0000", 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9731000000000002e-05 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:64703 3.e 3:74be2054:::rbd_data.112d420e934f.0000000000000007:21 [read 6938624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.562989+0000", 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.7969518400000002, 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038408454000000002, 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 64703 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562989+0000", 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562989+0000", 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562990+0000", 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562992+0000", 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1639999999999999e-06 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.562998+0000", 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.711e-06 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.563001+0000", 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7069999999999998e-06 2024-09-16T13:18:19.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601303+0000", 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038301833 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601339+0000", 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5377999999999999e-05 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.601398+0000", 2024-09-16T13:18:19.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9281999999999999e-05 2024-09-16T13:18:19.458 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.458 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.458 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.458 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.458 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.458 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:19.459 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone34 from 0xdd00eeb (overlap 0xdd00eeb) to 0xb0cd8bc 2024-09-16T13:18:19.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:19.460 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:19.451+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:19.560 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:19.556+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:19.571 INFO:teuthology.orchestra.run.smithi151.stdout:get_pg_num_history pg_num_history pg_num_history(e81 pg_nums {1={15=1},2={18=8},3={23=32}} deleted_pools ) 2024-09-16T13:18:19.571 INFO:teuthology.orchestra.run.smithi151.stdout:Importing pgid 3.15 2024-09-16T13:18:19.572 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:a8008361:::rbd_data.112de4fe95f7.00000000000000a9:a# 2024-09-16T13:18:19.573 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:19.573 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:19.574 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:a8008361:::rbd_data.112de4fe95f7.00000000000000a9:head# 2024-09-16T13:18:19.574 INFO:teuthology.orchestra.run.smithi151.stdout:snapset a=[]:{a=[a]} 2024-09-16T13:18:19.575 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:a8246e5d:::rbd_data.112de4fe95f7.000000000000002f:head# 2024-09-16T13:18:19.575 INFO:teuthology.orchestra.run.smithi151.stdout:snapset a=[]:{} 2024-09-16T13:18:19.575 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:a88a7d7c:::rbd_data.112ddef65b5a.0000000000000026:b# 2024-09-16T13:18:19.577 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:a88a7d7c:::rbd_data.112ddef65b5a.0000000000000026:head# 2024-09-16T13:18:19.577 INFO:teuthology.orchestra.run.smithi151.stdout:snapset b=[]:{b=[b]} 2024-09-16T13:18:19.577 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:a88e3a25:::rbd_data.112d7dbb182b.000000000000000d:head# 2024-09-16T13:18:19.577 INFO:teuthology.orchestra.run.smithi151.stdout:snapset 0=[]:{} 2024-09-16T13:18:19.578 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aa21e89a:::rbd_data.112de4fe95f7.0000000000000083:a# 2024-09-16T13:18:19.579 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aa21e89a:::rbd_data.112de4fe95f7.0000000000000083:head# 2024-09-16T13:18:19.579 INFO:teuthology.orchestra.run.smithi151.stdout:snapset a=[]:{a=[a]} 2024-09-16T13:18:19.579 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aa2d3911:::rbd_data.112de3ebec3a.000000000000004c:head# 2024-09-16T13:18:19.579 INFO:teuthology.orchestra.run.smithi151.stdout:snapset 0=[]:{} 2024-09-16T13:18:19.580 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aa4fd0df:::rbd_data.112d7dbb182b.00000000000001d5:head# 2024-09-16T13:18:19.580 INFO:teuthology.orchestra.run.smithi151.stdout:snapset 0=[]:{} 2024-09-16T13:18:19.581 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aa8bad7a:::rbd_data.112de4fe95f7.000000000000004c:a# 2024-09-16T13:18:19.581 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aa8bad7a:::rbd_data.112de4fe95f7.000000000000004c:head# 2024-09-16T13:18:19.582 INFO:teuthology.orchestra.run.smithi151.stdout:snapset a=[]:{a=[a]} 2024-09-16T13:18:19.582 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aaac9709:::rbd_data.112de4fe95f7.000000000000001f:a# 2024-09-16T13:18:19.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:19.584 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aaac9709:::rbd_data.112de4fe95f7.000000000000001f:head# 2024-09-16T13:18:19.585 INFO:teuthology.orchestra.run.smithi151.stdout:snapset a=[]:{a=[a]} 2024-09-16T13:18:19.585 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aad07d71:::rbd_data.112d8b6bb15.000000000000003e:head# 2024-09-16T13:18:19.585 INFO:teuthology.orchestra.run.smithi151.stdout:snapset 12=[]:{} 2024-09-16T13:18:19.585 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:ab32064f:::rbd_data.112d2ce070e7.0000000000000017:head# 2024-09-16T13:18:19.585 INFO:teuthology.orchestra.run.smithi151.stdout:snapset 5=[]:{} 2024-09-16T13:18:19.586 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:ab5f8439:::rbd_data.112db432a9c2.0000000000000019:head# 2024-09-16T13:18:19.586 INFO:teuthology.orchestra.run.smithi151.stdout:snapset 17=[]:{} 2024-09-16T13:18:19.586 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:ac588cdb:::rbd_data.112dfb79b81.0000000000000006:1a# 2024-09-16T13:18:19.589 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:ac588cdb:::rbd_data.112dfb79b81.0000000000000006:head# 2024-09-16T13:18:19.589 INFO:teuthology.orchestra.run.smithi151.stdout:snapset 1a=[]:{1a=[1a]} 2024-09-16T13:18:19.591 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:ad321648:::rbd_data.112de4fe95f7.0000000000000014:head# 2024-09-16T13:18:19.591 INFO:teuthology.orchestra.run.smithi151.stdout:snapset a=[]:{} 2024-09-16T13:18:19.591 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:head# 2024-09-16T13:18:19.593 INFO:teuthology.orchestra.run.smithi151.stdout:snapset 0=[]:{} 2024-09-16T13:18:19.600 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:ae604b58:::rbd_data.112ddef65b5a.000000000000001a:head# 2024-09-16T13:18:19.600 INFO:teuthology.orchestra.run.smithi151.stdout:snapset b=[]:{} 2024-09-16T13:18:19.600 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:ae74f048:::rbd_data.112d420e934f.0000000000000002:head# 2024-09-16T13:18:19.602 INFO:teuthology.orchestra.run.smithi151.stdout:snapset 0=[]:{} 2024-09-16T13:18:19.605 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aecdbb92:::rbd_data.112df1cc59bd.0000000000000013:d# 2024-09-16T13:18:19.608 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aecdbb92:::rbd_data.112df1cc59bd.0000000000000013:head# 2024-09-16T13:18:19.608 INFO:teuthology.orchestra.run.smithi151.stdout:snapset d=[]:{d=[d]} 2024-09-16T13:18:19.609 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aeedf058:::rbd_object_map.112d58339818.000000000000001d:head# 2024-09-16T13:18:19.609 INFO:teuthology.orchestra.run.smithi151.stdout:snapset 0=[]:{} 2024-09-16T13:18:19.612 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:aeffa2da:::rbd_data.112de4fe95f7.0000000000000041:head# 2024-09-16T13:18:19.612 INFO:teuthology.orchestra.run.smithi151.stdout:snapset a=[]:{} 2024-09-16T13:18:19.612 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:afdb193a:::rbd_data.112de4fe95f7.00000000000000a7:a# 2024-09-16T13:18:19.614 INFO:teuthology.orchestra.run.smithi151.stdout:Write #3:afdb193a:::rbd_data.112de4fe95f7.00000000000000a7:head# 2024-09-16T13:18:19.615 INFO:teuthology.orchestra.run.smithi151.stdout:snapset a=[]:{a=[a]} 2024-09-16T13:18:19.615 INFO:teuthology.orchestra.run.smithi151.stdout:write_pg epoch 80 info 3.15( v 80'162 (0'0,80'162] local-lis/les=23/24 n=29 ec=23/23 lis/c=23/23 les/c/f=24/24/0 sis=23) 2024-09-16T13:18:19.617 INFO:teuthology.orchestra.run.smithi151.stdout:Import successful 2024-09-16T13:18:19.661 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:19.656+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:19.670 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:19.670 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:19.670 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:19.671 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:19.671 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:19.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:19.748 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #33, image image_client.0-clone33 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent34 2024-09-16T13:18:19.762 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:19.756+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:19.781 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:19.781 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:19.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:19.781 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:19.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.781 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:18:19.781 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:19.781 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.953106881, 2024-09-16T13:18:19.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:18:19.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:18:19.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:18:19.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75103 3.1d 3:ba38d27a:::rbd_header.112d9326a5dd:head [watch reconnect cookie 140083112227968 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.434558+0000", 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34102048699999998, 2024-09-16T13:18:19.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043398552999999999, 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75103 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434558+0000", 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434558+0000", 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434559+0000", 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7399999999999991e-07 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434561+0000", 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4269999999999998e-06 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434567+0000", 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7370000000000003e-06 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434570+0000", 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.495e-06 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.468996+0000", 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.034425509999999999 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.469084+0000", 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7750999999999994e-05 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477956+0000", 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0088726589999999998 2024-09-16T13:18:19.792 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75235 3.1d 3:babb8944:::rbd_data.112d3c2065c6.000000000000000b:25 [read 1441792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435173+0000", 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34040486599999997, 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040680301000000002, 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.793 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75235 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435173+0000", 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435173+0000", 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435174+0000", 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435175+0000", 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6750000000000001e-06 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435181+0000", 2024-09-16T13:18:19.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0240000000000003e-06 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435186+0000", 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7310000000000001e-06 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.475667+0000", 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040481348 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.475761+0000", 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3919999999999995e-05 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.475854+0000", 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2215999999999999e-05 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.797 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75236 3.1d 3:babb8944:::rbd_data.112d3c2065c6.000000000000000b:25 [read 1769472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435192+0000", 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.340386087, 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040875799999999997, 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75236 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435192+0000", 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435192+0000", 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435192+0000", 2024-09-16T13:18:19.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8400000000000005e-07 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435194+0000", 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9259999999999999e-06 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435201+0000", 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5640000000000002e-06 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435204+0000", 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1829999999999998e-06 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.475889+0000", 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040684413000000003 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.475986+0000", 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7050000000000001e-05 2024-09-16T13:18:19.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476068+0000", 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2180000000000003e-05 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75237 3.1d 3:babb8944:::rbd_data.112d3c2065c6.000000000000000b:25 [read 2064384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435208+0000", 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34036987400000002, 2024-09-16T13:18:19.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041996782000000003, 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75237 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435208+0000", 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435208+0000", 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435209+0000", 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435210+0000", 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8080000000000001e-06 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435216+0000", 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3539999999999999e-06 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435219+0000", 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3419999999999999e-06 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477065+0000", 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041845787000000002 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477137+0000", 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2014000000000005e-05 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477205+0000", 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8089999999999994e-05 2024-09-16T13:18:19.806 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75274 3.d 3:b0e641d3:::rbd_data.112d3c2065c6.0000000000000019:25 [read 196608~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435223+0000", 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34035553000000002, 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041113480000000001, 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.807 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75274 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435223+0000", 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435223+0000", 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435223+0000", 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9900000000000001e-07 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435225+0000", 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.751e-06 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435230+0000", 2024-09-16T13:18:19.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5779999999999998e-06 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435239+0000", 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9930000000000001e-06 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476098+0000", 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040858271000000002 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476188+0000", 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0575000000000006e-05 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476336+0000", 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147913 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75275 3.d 3:b0e641d3:::rbd_data.112d3c2065c6.0000000000000019:25 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.811 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435253+0000", 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.340325088, 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041335887000000002, 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75275 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435253+0000", 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435253+0000", 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435254+0000", 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-16T13:18:19.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435261+0000", 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435267+0000", 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7320000000000001e-06 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435275+0000", 2024-09-16T13:18:19.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5020000000000003e-06 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476359+0000", 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041084260999999997 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476442+0000", 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3015000000000004e-05 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476589+0000", 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014677700000000001 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75276 3.d 3:b0e641d3:::rbd_data.112d3c2065c6.0000000000000019:25 [read 1146880~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435279+0000", 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34029923200000001, 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041563015000000002, 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.816 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75276 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435279+0000", 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435279+0000", 2024-09-16T13:18:19.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435279+0000", 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435281+0000", 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435287+0000", 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435290+0000", 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2440000000000002e-06 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476615+0000", 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041324913999999997 2024-09-16T13:18:19.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476702+0000", 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7394999999999999e-05 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476842+0000", 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013951499999999999 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75277 3.d 3:b0e641d3:::rbd_data.112d3c2065c6.0000000000000019:25 [read 1441792~31893] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435294+0000", 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34028434899999999, 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042150939999999998, 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75277 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435294+0000", 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435294+0000", 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435294+0000", 2024-09-16T13:18:19.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7599999999999998e-07 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435296+0000", 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7400000000000001e-06 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435303+0000", 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6789999999999999e-06 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435306+0000", 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.281e-06 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476884+0000", 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041577792000000002 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476950+0000", 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6031999999999994e-05 2024-09-16T13:18:19.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477445+0000", 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049503999999999998 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75278 3.d 3:b0e641d3:::rbd_data.112d3c2065c6.0000000000000019:25 [read 1769472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435310+0000", 2024-09-16T13:18:19.825 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34026816900000001, 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042650988000000001, 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75278 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435310+0000", 2024-09-16T13:18:19.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435310+0000", 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435310+0000", 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435312+0000", 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.844e-06 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435318+0000", 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6949999999999997e-06 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435321+0000", 2024-09-16T13:18:19.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.382e-06 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477501+0000", 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042179609 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477660+0000", 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015907700000000001 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477961+0000", 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030097499999999998 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75313 3.1d 3:be4d3b50:::rbd_data.112d3c2065c6.0000000000000028:25 [read 196608~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435446+0000", 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34013195200000002, 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042256112999999998, 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.830 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75313 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435446+0000", 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435446+0000", 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435447+0000", 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435448+0000", 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7990000000000001e-06 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435453+0000", 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0490000000000003e-06 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435457+0000", 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1190000000000001e-06 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477469+0000", 2024-09-16T13:18:19.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042012723000000002 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477558+0000", 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8477e-05 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477702+0000", 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014455800000000001 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75314 3.1d 3:be4d3b50:::rbd_data.112d3c2065c6.0000000000000028:25 [read 819200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435461+0000", 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34011743700000002, 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042415044999999998, 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75314 2024-09-16T13:18:19.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435461+0000", 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435461+0000", 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435461+0000", 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8800000000000003e-07 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435463+0000", 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9080000000000002e-06 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435469+0000", 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7660000000000004e-06 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435472+0000", 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1659999999999998e-06 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477725+0000", 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042253023000000001 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477801+0000", 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6370000000000002e-05 2024-09-16T13:18:19.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477876+0000", 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4324000000000002e-05 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75315 3.1d 3:be4d3b50:::rbd_data.112d3c2065c6.0000000000000028:25 [read 1146880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435476+0000", 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34010259399999998, 2024-09-16T13:18:19.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043258913000000003, 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75315 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435476+0000", 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435476+0000", 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435476+0000", 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435478+0000", 2024-09-16T13:18:19.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7260000000000001e-06 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435483+0000", 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435488+0000", 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2230000000000001e-06 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478020+0000", 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042532055999999999 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478590+0000", 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057009699999999999 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478734+0000", 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014470800000000001 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.843 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75316 3.1d 3:be4d3b50:::rbd_data.112d3c2065c6.0000000000000028:25 [read 1441792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435491+0000", 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34008704000000001, 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043410124000000001, 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75316 2024-09-16T13:18:19.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435491+0000", 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435491+0000", 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435491+0000", 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8099999999999998e-07 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435493+0000", 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7e-06 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435499+0000", 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4110000000000004e-06 2024-09-16T13:18:19.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435502+0000", 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4620000000000001e-06 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478762+0000", 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043260028999999998 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478822+0000", 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0204999999999997e-05 2024-09-16T13:18:19.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478901+0000", 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8936000000000005e-05 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75317 3.1d 3:be4d3b50:::rbd_data.112d3c2065c6.0000000000000028:25 [read 1769472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435505+0000", 2024-09-16T13:18:19.848 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.340072614, 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044500790999999998, 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75317 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435505+0000", 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435505+0000", 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435506+0000", 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8500000000000002e-07 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435507+0000", 2024-09-16T13:18:19.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4839999999999999e-06 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435513+0000", 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2920000000000003e-06 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435516+0000", 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5439999999999999e-06 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479903+0000", 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044387175000000001 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479955+0000", 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1721999999999997e-05 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.480006+0000", 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1088999999999998e-05 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75382 3.d 3:b6152032:::rbd_data.112d3c2065c6.0000000000000048:25 [read 491520~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435520+0000", 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34005800800000002, 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043630454999999999, 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75382 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435520+0000", 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435520+0000", 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435520+0000", 2024-09-16T13:18:19.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435522+0000", 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435528+0000", 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1880000000000001e-06 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435531+0000", 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999999e-06 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478928+0000", 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043397133999999997 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479017+0000", 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8769000000000003e-05 2024-09-16T13:18:19.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479151+0000", 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013375600000000001 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75383 3.d 3:b6152032:::rbd_data.112d3c2065c6.0000000000000048:25 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435540+0000", 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34003789400000001, 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043809694000000003, 2024-09-16T13:18:19.857 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75383 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435540+0000", 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435540+0000", 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435541+0000", 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435551+0000", 2024-09-16T13:18:19.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0457e-05 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435559+0000", 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4669999999999998e-06 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435562+0000", 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479183+0000", 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043620402000000003 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479274+0000", 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1289e-05 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479350+0000", 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5862e-05 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.861 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75384 3.d 3:b6152032:::rbd_data.112d3c2065c6.0000000000000048:25 [read 1114112~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435567+0000", 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34001130000000002, 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043967154000000001, 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.862 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75384 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435567+0000", 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435567+0000", 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435567+0000", 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435569+0000", 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9750000000000001e-06 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435575+0000", 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5659999999999997e-06 2024-09-16T13:18:19.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435583+0000", 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.473e-06 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479377+0000", 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043793374000000003 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479433+0000", 2024-09-16T13:18:19.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6422000000000003e-05 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479534+0000", 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100929 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75385 3.d 3:b6152032:::rbd_data.112d3c2065c6.0000000000000048:25 [read 1441792~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435588+0000", 2024-09-16T13:18:19.866 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.33999038100000001, 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044113678000000003, 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75385 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435588+0000", 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435588+0000", 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435588+0000", 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2399999999999999e-07 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435590+0000", 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.119e-06 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435598+0000", 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8949999999999998e-06 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435604+0000", 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7760000000000003e-06 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479557+0000", 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043952705000000002 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479606+0000", 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9305999999999999e-05 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479701+0000", 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5452999999999996e-05 2024-09-16T13:18:19.870 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75386 3.d 3:b6152032:::rbd_data.112d3c2065c6.0000000000000048:25 [read 2064384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435608+0000", 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.33997004600000003, 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044265631, 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:19.871 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75386 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435608+0000", 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435608+0000", 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435608+0000", 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3599999999999999e-07 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435611+0000", 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1689999999999999e-06 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435618+0000", 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7270000000000007e-06 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435628+0000", 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1880000000000008e-06 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479724+0000", 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044096718999999999 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:19.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479770+0000", 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5274000000000001e-05 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479874+0000", 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010411799999999999 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:19.875 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:19.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:19.877 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:19.872+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:19.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:19.978 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:19.972+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:19.979 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:19.979 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:19.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:20.011 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:20.029 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:20.048 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:20.065 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:20.079 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:20.076+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:20.088 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:20.091 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:20.092 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:20.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:20.110 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:20.140 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:20.141 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:20.141 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:20.141 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:20.141 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:20.141 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:20.141 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:20.141 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:20.142 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:20.144 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:20.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /var/log/ceph/exp.3.15.7 2024-09-16T13:18:20.173 INFO:teuthology.orchestra.run.smithi098.stdout:4727 read 0x8c25500 thru 0x8c28ec5 (0x39c6 bytes) 2024-09-16T13:18:20.176 INFO:teuthology.orchestra.run.smithi098.stdout:4728 punch from 0x65688a1 to 0x656bca9, (0x3408 bytes) 2024-09-16T13:18:20.179 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:20.176+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:20.182 INFO:teuthology.orchestra.run.smithi098.stdout:4729 read 0x5e6b308 thru 0x5e740cd (0x8dc6 bytes) 2024-09-16T13:18:20.184 INFO:teuthology.orchestra.run.smithi098.stdout:4730 read 0x28b7139 thru 0x28c4082 (0xcf4a bytes) 2024-09-16T13:18:20.185 INFO:teuthology.orchestra.run.smithi098.stdout:4731 write 0x9928c00 thru 0x992a13c (0x153d bytes) 2024-09-16T13:18:20.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:18:20.201 INFO:teuthology.orchestra.run.smithi098.stdout:4732 write 0x13ded62 thru 0x13e8990 (0x9c2f bytes) 2024-09-16T13:18:20.202 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:20.202 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:20.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:20.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:18:20.252 INFO:teuthology.orchestra.run.smithi098.stdout:4733 trunc from 0xdd00eeb to 0xbfe74ef 2024-09-16T13:18:20.280 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:20.271+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:20.290 INFO:teuthology.orchestra.run.smithi098.stdout:4734 punch from 0x54e2964 to 0x54e5704, (0x2da0 bytes) 2024-09-16T13:18:20.290 INFO:teuthology.orchestra.run.smithi098.stdout:4735 punch from 0x6ee2e3e to 0x6ef1ea6, (0xf068 bytes) 2024-09-16T13:18:20.304 INFO:teuthology.orchestra.run.smithi098.stdout:4736 write 0xe84858b thru 0xe849611 (0x1087 bytes) 2024-09-16T13:18:20.321 INFO:teuthology.orchestra.run.smithi098.stdout:4737 trunc from 0xe849612 to 0x737fa51 2024-09-16T13:18:20.329 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:20.330 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:20.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:20.381 INFO:teuthology.orchestra.run.smithi098.stdout:4738 trunc from 0x737fa51 to 0x38e0b09 2024-09-16T13:18:20.381 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:20.375+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:20.482 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:20.476+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:20.499 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:20.500 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:20.509 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:20.525 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:18:20.525 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":88,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:18:19.221148+0000","last_up_change":"2024-09-16T13:18:10.165969+0000","last_in_change":"2024-09-16T13:17:42.548353+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:16:33.497561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:16:40.994538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-16T13:16:50.759705+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":38,"snap_epoch":88,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":1,"in":1,"weight":0.8320770263671875,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":22271},{"type":"v1","addr":"172.21.15.98:6809","nonce":22271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":22271},{"type":"v1","addr":"172.21.15.98:6811","nonce":22271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":22271},{"type":"v1","addr":"172.21.15.98:6817","nonce":22271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":22271},{"type":"v1","addr":"172.21.15.98:6814","nonce":22271}]},"public_addr":"172.21.15.98:6809/22271","cluster_addr":"172.21.15.98:6811/22271","heartbeat_back_addr":"172.21.15.98:6817/22271","heartbeat_front_addr":"172.21.15.98:6814/22271","state":["exists","up"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":22272},{"type":"v1","addr":"172.21.15.98:6815","nonce":22272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":22272},{"type":"v1","addr":"172.21.15.98:6819","nonce":22272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":22272},{"type":"v1","addr":"172.21.15.98:6823","nonce":22272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":22272},{"type":"v1","addr":"172.21.15.98:6821","nonce":22272}]},"public_addr":"172.21.15.98:6815/22272","cluster_addr":"172.21.15.98:6819/22272","heartbeat_back_addr":"172.21.15.98:6823/22272","heartbeat_front_addr":"172.21.15.98:6821/22272","state":["exists","up"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":22273},{"type":"v1","addr":"172.21.15.98:6801","nonce":22273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":22273},{"type":"v1","addr":"172.21.15.98:6803","nonce":22273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":22273},{"type":"v1","addr":"172.21.15.98:6807","nonce":22273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":22273},{"type":"v1","addr":"172.21.15.98:6805","nonce":22273}]},"public_addr":"172.21.15.98:6801/22273","cluster_addr":"172.21.15.98:6803/22273","heartbeat_back_addr":"172.21.15.98:6807/22273","heartbeat_front_addr":"172.21.15.98:6805/22273","state":["exists","up"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":22270},{"type":"v1","addr":"172.21.15.98:6825","nonce":22270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":22270},{"type":"v1","addr":"172.21.15.98:6827","nonce":22270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":22270},{"type":"v1","addr":"172.21.15.98:6831","nonce":22270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":22270},{"type":"v1","addr":"172.21.15.98:6829","nonce":22270}]},"public_addr":"172.21.15.98:6825/22270","cluster_addr":"172.21.15.98:6827/22270","heartbeat_back_addr":"172.21.15.98:6831/22270","heartbeat_front_addr":"172.21.15.98:6829/22270","state":["exists","up"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":22286},{"type":"v1","addr":"172.21.15.151:6825","nonce":22286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":22286},{"type":"v1","addr":"172.21.15.151:6827","nonce":22286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":22286},{"type":"v1","addr":"172.21.15.151:6831","nonce":22286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":22286},{"type":"v1","addr":"172.21.15.151:6829","nonce":22286}]},"public_addr":"172.21.15.151:6825/22286","cluster_addr":"172.21.15.151:6827/22286","heartbeat_back_addr":"172.21.15.151:6831/22286","heartbeat_front_addr":"172.21.15.151:6829/22286","state":["exists","up"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":22285},{"type":"v1","addr":"172.21.15.151:6809","nonce":22285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":22285},{"type":"v1","addr":"172.21.15.151:6811","nonce":22285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":22285},{"type":"v1","addr":"172.21.15.151:6815","nonce":22285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":22285},{"type":"v1","addr":"172.21.15.151:6813","nonce":22285}]},"public_addr":"172.21.15.151:6809/22285","cluster_addr":"172.21.15.151:6811/22285","heartbeat_back_addr":"172.21.15.151:6815/22285","heartbeat_front_addr":"172.21.15.151:6813/22285","state":["exists","up"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":22315},{"type":"v1","addr":"172.21.15.151:6817","nonce":22315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":22315},{"type":"v1","addr":"172.21.15.151:6819","nonce":22315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":22315},{"type":"v1","addr":"172.21.15.151:6823","nonce":22315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":22315},{"type":"v1","addr":"172.21.15.151:6821","nonce":22315}]},"public_addr":"172.21.15.151:6817/22315","cluster_addr":"172.21.15.151:6819/22315","heartbeat_back_addr":"172.21.15.151:6823/22315","heartbeat_front_addr":"172.21.15.151:6821/22315","state":["exists","up"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":22284},{"type":"v1","addr":"172.21.15.151:6801","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":22284},{"type":"v1","addr":"172.21.15.151:6803","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":22284},{"type":"v1","addr":"172.21.15.151:6807","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":22284},{"type":"v1","addr":"172.21.15.151:6805","nonce":22284}]},"public_addr":"172.21.15.151:6801/22284","cluster_addr":"172.21.15.151:6803/22284","heartbeat_back_addr":"172.21.15.151:6807/22284","heartbeat_front_addr":"172.21.15.151:6805/22284","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.191311+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.563551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.493457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.599815+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.634352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.262873+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.508881+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.348276+0000","dead_epoch":82}],"pg_upmap":[{"pgid":"3.1c","osds":[6,1]},{"pgid":"3.1f","osds":[5,3]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:18:20.541 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T13:16:33.497561+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T13:16:40.994538+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 21, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-16T13:16:50.759705+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 38, 'snap_epoch': 88, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-16T13:18:20.541 DEBUG:teuthology.orchestra.run.smithi151:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op apply-layout-settings --pool .mgr 2024-09-16T13:18:20.583 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:20.580+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:20.603 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:20.603 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:20.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:20.657 INFO:teuthology.orchestra.run.smithi098.stdout:4739 punch from 0x2725ba to 0x273260, (0xca6 bytes) 2024-09-16T13:18:20.679 INFO:teuthology.orchestra.run.smithi098.stdout:4740 write 0xa9add80 thru 0xa9b510d (0x738e bytes) 2024-09-16T13:18:20.684 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:20.680+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:20.690 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:4742 trunc from 0xa9b510e to 0xe126a74 2024-09-16T13:18:20.705 INFO:teuthology.orchestra.run.smithi098.stdout:4743 write 0xd8596d9 thru 0xd863c65 (0xa58d bytes) 2024-09-16T13:18:20.713 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:20.713 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:20.716 INFO:teuthology.orchestra.run.smithi098.stdout:4744 read 0xa62eeba thru 0xa636f63 (0x80aa bytes) 2024-09-16T13:18:20.717 INFO:teuthology.orchestra.run.smithi098.stdout:4745 writesame 0x5facb0a thru 0x5fb0425 data_size 0x44(0x391c bytes) 2024-09-16T13:18:20.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:4747 read 0x5278384 thru 0x527cdbf (0x4a3c bytes) 2024-09-16T13:18:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:4748 punch from 0xae2c11d to 0xae38d91, (0xcc74 bytes) 2024-09-16T13:18:20.724 INFO:teuthology.orchestra.run.smithi098.stdout:4749 write 0x662d62f thru 0x663c57b (0xef4d bytes) 2024-09-16T13:18:20.734 INFO:teuthology.orchestra.run.smithi098.stdout:4750 writesame 0x7d0669f thru 0x7d13e1f data_size 0x1(0xd781 bytes) 2024-09-16T13:18:20.782 INFO:teuthology.orchestra.run.smithi098.stdout:4751 read 0x6edaee8 thru 0x6ee3363 (0x847c bytes) 2024-09-16T13:18:20.785 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:20.779+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:20.802 INFO:teuthology.orchestra.run.smithi098.stdout:4752 trunc from 0xe126a74 to 0x1c8039d 2024-09-16T13:18:20.826 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:20.827 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:20.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:20.886 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:20.879+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:20.900 INFO:teuthology.orchestra.run.smithi098.stdout:4753 write 0xbe323a9 thru 0xbe3d78a (0xb3e2 bytes) 2024-09-16T13:18:20.918 INFO:teuthology.orchestra.run.smithi098.stdout:4754 trunc from 0xbe3d78b to 0xe0d1837 2024-09-16T13:18:20.927 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:20.927 INFO:teuthology.orchestra.run.smithi098.stdout:4756 read 0xae7e7b4 thru 0xae88c45 (0xa492 bytes) 2024-09-16T13:18:20.931 INFO:teuthology.orchestra.run.smithi098.stdout:4757 trunc from 0xe0d1837 to 0xbfb85af 2024-09-16T13:18:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:4763 read 0xbe92133 thru 0xbe9c511 (0xa3df bytes) 2024-09-16T13:18:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:4765 read 0x4919822 thru 0x491cfd0 (0x37af bytes) 2024-09-16T13:18:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:4766 read 0xcd7f50 thru 0xce10de (0x918f bytes) 2024-09-16T13:18:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:4768 write 0x98d86f thru 0x998d7f (0xb511 bytes) 2024-09-16T13:18:20.987 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:20.979+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:21.008 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:21.008 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:21.008 INFO:teuthology.orchestra.run.smithi098.stdout:4769 writesame 0xe0b90b1 thru 0xe0bcefa data_size 0x43(0x3e4a bytes) 2024-09-16T13:18:21.009 INFO:teuthology.orchestra.run.smithi098.stdout:4771 write 0x9ee7d39 thru 0x9ef373f (0xba07 bytes) 2024-09-16T13:18:21.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:21.020 INFO:teuthology.orchestra.run.smithi098.stdout:4772 punch from 0xc392dd9 to 0xc39aece, (0x80f5 bytes) 2024-09-16T13:18:21.020 INFO:teuthology.orchestra.run.smithi098.stdout:4774 writesame 0x6533c3b thru 0x653e4c9 data_size 0x1(0xa88f bytes) 2024-09-16T13:18:21.056 INFO:teuthology.orchestra.run.smithi098.stdout:4775 writesame 0xd24d22 thru 0xd26aca data_size 0x1(0x1da9 bytes) 2024-09-16T13:18:21.064 INFO:teuthology.orchestra.run.smithi098.stdout:4776 read 0x45256d6 thru 0x45292ee (0x3c19 bytes) 2024-09-16T13:18:21.064 INFO:teuthology.orchestra.run.smithi098.stdout:4777 write 0xa34990c thru 0xa3516a8 (0x7d9d bytes) 2024-09-16T13:18:21.073 INFO:teuthology.orchestra.run.smithi098.stdout:4779 trunc from 0xe0bcefb to 0xc415257 2024-09-16T13:18:21.088 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:21.084+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:21.089 INFO:teuthology.orchestra.run.smithi098.stdout:4780 trunc from 0xc415257 to 0xe57c892 2024-09-16T13:18:21.100 INFO:teuthology.orchestra.run.smithi098.stdout:4781 write 0x2c9d202 thru 0x2ca2b1d (0x591c bytes) 2024-09-16T13:18:21.109 INFO:teuthology.orchestra.run.smithi098.stdout:4782 punch from 0x72166a4 to 0x72184f1, (0x1e4d bytes) 2024-09-16T13:18:21.109 INFO:teuthology.orchestra.run.smithi098.stdout:4783 write 0x4b0ac7e thru 0x4b14b5f (0x9ee2 bytes) 2024-09-16T13:18:21.120 INFO:teuthology.orchestra.run.smithi098.stdout:4785 read 0xacaaa7e thru 0xacb0045 (0x55c8 bytes) 2024-09-16T13:18:21.120 INFO:teuthology.orchestra.run.smithi098.stdout:4786 writesame 0xbc975e7 thru 0xbc9e762 data_size 0x36(0x717c bytes) 2024-09-16T13:18:21.127 INFO:teuthology.orchestra.run.smithi098.stdout:4788 read 0xa38d15d thru 0xa399cac (0xcb50 bytes) 2024-09-16T13:18:21.127 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:21.127 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:21.128 INFO:teuthology.orchestra.run.smithi098.stdout:4789 read 0xd6446a0 thru 0xd646187 (0x1ae8 bytes) 2024-09-16T13:18:21.128 INFO:teuthology.orchestra.run.smithi098.stdout:4790 write 0x678e3c9 thru 0x679d8b6 (0xf4ee bytes) 2024-09-16T13:18:21.134 INFO:teuthology.orchestra.run.smithi098.stdout:4791 read 0xcaa4723 thru 0xcab150d (0xcdeb bytes) 2024-09-16T13:18:21.135 INFO:teuthology.orchestra.run.smithi098.stdout:4792 writesame 0x3de29f4 thru 0x3deb87c data_size 0x1(0x8e89 bytes) 2024-09-16T13:18:21.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:21.170 INFO:teuthology.orchestra.run.smithi098.stdout:4793 punch from 0x8320a9b to 0x832314a, (0x26af bytes) 2024-09-16T13:18:21.170 INFO:teuthology.orchestra.run.smithi098.stdout:4794 write 0xb9366f6 thru 0xb94210f (0xba1a bytes) 2024-09-16T13:18:21.175 INFO:teuthology.orchestra.run.smithi098.stdout:4795 punch from 0x59540cc to 0x5954414, (0x348 bytes) 2024-09-16T13:18:21.175 INFO:teuthology.orchestra.run.smithi098.stdout:4796 punch from 0x8ba401c to 0x8baecee, (0xacd2 bytes) 2024-09-16T13:18:21.175 INFO:teuthology.orchestra.run.smithi098.stdout:4797 write 0x688622 thru 0x68bf94 (0x3973 bytes) 2024-09-16T13:18:21.189 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:21.183+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:21.212 INFO:teuthology.orchestra.run.smithi098.stdout:4798 trunc from 0xe57c892 to 0x639d66e 2024-09-16T13:18:21.242 INFO:teuthology.orchestra.run.smithi098.stdout:4799 write 0x15f482a thru 0x15f722e (0x2a05 bytes) 2024-09-16T13:18:21.246 INFO:teuthology.orchestra.run.smithi098.stdout:4800 punch from 0x17e5050 to 0x17e8908, (0x38b8 bytes) 2024-09-16T13:18:21.246 INFO:teuthology.orchestra.run.smithi098.stdout:4802 write 0x197b45b thru 0x19807aa (0x5350 bytes) 2024-09-16T13:18:21.252 INFO:teuthology.orchestra.run.smithi098.stdout:4803 writesame 0x5f6cdf1 thru 0x5f6f003 data_size 0xd(0x2213 bytes) 2024-09-16T13:18:21.259 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:18:21.262 INFO:teuthology.orchestra.run.smithi098.stdout:4804 read 0x5258f46 thru 0x5259174 (0x22f bytes) 2024-09-16T13:18:21.263 INFO:teuthology.orchestra.run.smithi098.stdout:4805 write 0x3616b78 thru 0x36215ae (0xaa37 bytes) 2024-09-16T13:18:21.273 INFO:teuthology.orchestra.run.smithi098.stdout:4806 read 0x117f152 thru 0x118ce28 (0xdcd7 bytes) 2024-09-16T13:18:21.282 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:21.282 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:21.287 INFO:teuthology.orchestra.run.smithi098.stdout:4807 trunc from 0x639d66e to 0xe70c438 2024-09-16T13:18:21.290 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:21.283+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:21.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:21.296 INFO:teuthology.orchestra.run.smithi098.stdout:4808 read 0x240f2a4 thru 0x2413160 (0x3ebd bytes) 2024-09-16T13:18:21.296 INFO:teuthology.orchestra.run.smithi098.stdout:4809 writesame 0x86a2b6c thru 0x86a302a data_size 0xf(0x4bf bytes) 2024-09-16T13:18:21.302 INFO:teuthology.orchestra.run.smithi098.stdout:4810 read 0x75a6394 thru 0x75a74f2 (0x115f bytes) 2024-09-16T13:18:21.302 INFO:teuthology.orchestra.run.smithi098.stdout:4812 punch from 0xe3d6cbb to 0xe3d8bf1, (0x1f36 bytes) 2024-09-16T13:18:21.302 INFO:teuthology.orchestra.run.smithi098.stdout:4813 write 0xd915bcb thru 0xd919b72 (0x3fa8 bytes) 2024-09-16T13:18:21.310 INFO:teuthology.orchestra.run.smithi098.stdout:4814 write 0x99e23 thru 0x9e230 (0x440e bytes) 2024-09-16T13:18:21.329 INFO:teuthology.orchestra.run.smithi098.stdout:4817 write 0xc7cd7df thru 0xc7cebb5 (0x13d7 bytes) 2024-09-16T13:18:21.335 INFO:teuthology.orchestra.run.smithi098.stdout:4818 write 0xd2bf23a thru 0xd2c9ab9 (0xa880 bytes) 2024-09-16T13:18:21.346 INFO:teuthology.orchestra.run.smithi098.stdout:4819 punch from 0xd771679 to 0xd7734a9, (0x1e30 bytes) 2024-09-16T13:18:21.346 INFO:teuthology.orchestra.run.smithi098.stdout:4821 read 0x3cc6ec6 thru 0x3cce103 (0x723e bytes) 2024-09-16T13:18:21.347 INFO:teuthology.orchestra.run.smithi098.stdout:4822 write 0x7e0a642 thru 0x7e174ad (0xce6c bytes) 2024-09-16T13:18:21.356 INFO:teuthology.orchestra.run.smithi098.stdout:4824 read 0xd19c9ed thru 0xd1aa265 (0xd879 bytes) 2024-09-16T13:18:21.358 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.358 INFO:teuthology.orchestra.run.smithi098.stdout:4830 read 0xaf87651 thru 0xaf968a5 (0xf255 bytes) 2024-09-16T13:18:21.358 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.358 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.364 INFO:teuthology.orchestra.run.smithi098.stdout:4836 trunc from 0xe70c438 to 0xa4e05df 2024-09-16T13:18:21.385 INFO:teuthology.orchestra.run.smithi098.stdout:4837 punch from 0x274fdcf to 0x2755931, (0x5b62 bytes) 2024-09-16T13:18:21.385 INFO:teuthology.orchestra.run.smithi098.stdout:4838 read 0x8f73aa4 thru 0x8f7c22e (0x878b bytes) 2024-09-16T13:18:21.391 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:21.388+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:21.397 INFO:teuthology.orchestra.run.smithi098.stdout:4839 write 0xb46017e thru 0xb462e88 (0x2d0b bytes) 2024-09-16T13:18:21.399 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:21.399 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:21.404 INFO:teuthology.orchestra.run.smithi098.stdout:4840 writesame 0x7029b39 thru 0x70364e6 data_size 0xf(0xc9ae bytes) 2024-09-16T13:18:21.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:21.420 INFO:teuthology.orchestra.run.smithi098.stdout:4841 read 0x9ad3075 thru 0x9ad7a50 (0x49dc bytes) 2024-09-16T13:18:21.420 INFO:teuthology.orchestra.run.smithi098.stdout:4842 read 0x821b68a thru 0x82293d6 (0xdd4d bytes) 2024-09-16T13:18:21.420 INFO:teuthology.orchestra.run.smithi098.stdout:4843 read 0x986288a thru 0x9863c40 (0x13b7 bytes) 2024-09-16T13:18:21.428 INFO:teuthology.orchestra.run.smithi098.stdout:4844 write 0xba2ef17 thru 0xba33101 (0x41eb bytes) 2024-09-16T13:18:21.437 INFO:teuthology.orchestra.run.smithi098.stdout:4845 trunc from 0xba33102 to 0x96d3f13 2024-09-16T13:18:21.455 INFO:teuthology.orchestra.run.smithi098.stdout:4846 punch from 0x4a44b3e to 0x4a44f11, (0x3d3 bytes) 2024-09-16T13:18:21.455 INFO:teuthology.orchestra.run.smithi098.stdout:4847 read 0x2bea8e6 thru 0x2bf7474 (0xcb8f bytes) 2024-09-16T13:18:21.456 INFO:teuthology.orchestra.run.smithi098.stdout:4848 read 0x151f9dc thru 0x15218f6 (0x1f1b bytes) 2024-09-16T13:18:21.457 INFO:teuthology.orchestra.run.smithi098.stdout:4849 write 0x5d76385 thru 0x5d7a061 (0x3cdd bytes) 2024-09-16T13:18:21.465 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.465 INFO:teuthology.orchestra.run.smithi098.stdout:4852 read 0x4b93e6c thru 0x4ba0042 (0xc1d7 bytes) 2024-09-16T13:18:21.466 INFO:teuthology.orchestra.run.smithi098.stdout:4853 read 0x1405150 thru 0x1413d2d (0xebde bytes) 2024-09-16T13:18:21.467 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.467 INFO:teuthology.orchestra.run.smithi098.stdout:4855 write 0x31e8b18 thru 0x31ebc3f (0x3128 bytes) 2024-09-16T13:18:21.476 INFO:teuthology.orchestra.run.smithi098.stdout:4856 trunc from 0x96d3f13 to 0x66d0114 2024-09-16T13:18:21.492 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:21.488+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:21.501 INFO:teuthology.orchestra.run.smithi098.stdout:4859 read 0xc6b18a thru 0xc79d2a (0xeba1 bytes) 2024-09-16T13:18:21.503 INFO:teuthology.orchestra.run.smithi098.stdout:4860 read 0xc21977 thru 0xc308a1 (0xef2b bytes) 2024-09-16T13:18:21.504 INFO:teuthology.orchestra.run.smithi098.stdout:4861 read 0x281f936 thru 0x282b34e (0xba19 bytes) 2024-09-16T13:18:21.525 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:21.526 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:21.528 INFO:teuthology.orchestra.run.smithi098.stdout:4862 writesame 0xeaa2a99 thru 0xeaac6d0 data_size 0x4(0x9c38 bytes) 2024-09-16T13:18:21.528 INFO:teuthology.orchestra.run.smithi098.stdout:4863 writesame 0xdfa071b thru 0xdfa12d0 data_size 0x1(0xbb6 bytes) 2024-09-16T13:18:21.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:21.537 INFO:teuthology.orchestra.run.smithi098.stdout:4864 write 0x89479ee thru 0x8956768 (0xed7b bytes) 2024-09-16T13:18:21.551 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.551 INFO:teuthology.orchestra.run.smithi098.stdout:4866 read 0x47642f thru 0x476596 (0x168 bytes) 2024-09-16T13:18:21.552 INFO:teuthology.orchestra.run.smithi098.stdout:4867 write 0x41c23e4 thru 0x41c6690 (0x42ad bytes) 2024-09-16T13:18:21.557 INFO:teuthology.orchestra.run.smithi098.stdout:4868 writesame 0xbb7cab6 thru 0xbb7ce8c data_size 0x1(0x3d7 bytes) 2024-09-16T13:18:21.564 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.564 INFO:teuthology.orchestra.run.smithi098.stdout:4870 punch from 0xea5fbc4 to 0xea6494a, (0x4d86 bytes) 2024-09-16T13:18:21.564 INFO:teuthology.orchestra.run.smithi098.stdout:4873 write 0xcb19038 thru 0xcb27b8b (0xeb54 bytes) 2024-09-16T13:18:21.580 INFO:teuthology.orchestra.run.smithi098.stdout:4876 write 0xeb8d5f7 thru 0xeb8fef6 (0x2900 bytes) 2024-09-16T13:18:21.586 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.586 INFO:teuthology.orchestra.run.smithi098.stdout:4878 write 0x1e0f346 thru 0x1e16b74 (0x782f bytes) 2024-09-16T13:18:21.589 INFO:teuthology.orchestra.run.smithi098.stdout:4879 punch from 0x24bbc9d to 0x24c595a, (0x9cbd bytes) 2024-09-16T13:18:21.589 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.593 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:21.588+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:21.603 INFO:teuthology.orchestra.run.smithi098.stdout:4883 trunc from 0xeb8fef7 to 0x5b8374a 2024-09-16T13:18:21.625 INFO:teuthology.orchestra.run.smithi098.stdout:4884 read 0x393daa2 thru 0x39470ed (0x964c bytes) 2024-09-16T13:18:21.626 INFO:teuthology.orchestra.run.smithi098.stdout:4885 read 0x56fbdd5 thru 0x5705122 (0x934e bytes) 2024-09-16T13:18:21.644 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:21.644 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:21.646 INFO:teuthology.orchestra.run.smithi098.stdout:4888 write 0xdb765ee thru 0xdb77154 (0xb67 bytes) 2024-09-16T13:18:21.652 INFO:teuthology.orchestra.run.smithi098.stdout:4889 read 0x2a2bc3a thru 0x2a2d47c (0x1843 bytes) 2024-09-16T13:18:21.653 INFO:teuthology.orchestra.run.smithi098.stdout:4890 write 0xd45c4b9 thru 0xd465a5e (0x95a6 bytes) 2024-09-16T13:18:21.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:21.661 INFO:teuthology.orchestra.run.smithi098.stdout:4891 read 0xa7b8a48 thru 0xa7ba9bf (0x1f78 bytes) 2024-09-16T13:18:21.665 INFO:teuthology.orchestra.run.smithi098.stdout:4892 trunc from 0xdb77155 to 0xbe41296 2024-09-16T13:18:21.684 INFO:teuthology.orchestra.run.smithi098.stdout:4893 read 0x6f641 thru 0x7f29e (0xfc5e bytes) 2024-09-16T13:18:21.688 INFO:teuthology.orchestra.run.smithi098.stdout:4894 trunc from 0xbe41296 to 0x9f5d431 2024-09-16T13:18:21.694 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:21.688+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:21.705 INFO:teuthology.orchestra.run.smithi098.stdout:4895 trunc from 0x9f5d431 to 0xdd67774 2024-09-16T13:18:21.717 INFO:teuthology.orchestra.run.smithi098.stdout:4896 read 0x9721e2b thru 0x97274ef (0x56c5 bytes) 2024-09-16T13:18:21.717 INFO:teuthology.orchestra.run.smithi098.stdout:4897 read 0xbe43b7d thru 0xbe49277 (0x56fb bytes) 2024-09-16T13:18:21.717 INFO:teuthology.orchestra.run.smithi098.stdout:4899 punch from 0xc00261f to 0xc004608, (0x1fe9 bytes) 2024-09-16T13:18:21.718 INFO:teuthology.orchestra.run.smithi098.stdout:4900 writesame 0x444ddc8 thru 0x4454b6d data_size 0x5(0x6da6 bytes) 2024-09-16T13:18:21.732 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.732 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.745 INFO:teuthology.orchestra.run.smithi098.stdout:4903 trunc from 0xdd67774 to 0x5fee5db 2024-09-16T13:18:21.746 INFO:teuthology.orchestra.run.smithi151.stderr:Nothing to do for non-filestore backend 2024-09-16T13:18:21.758 INFO:teuthology.orchestra.run.smithi098.stdout:4904 read 0x2f6da39 thru 0x2f6dfc9 (0x591 bytes) 2024-09-16T13:18:21.779 INFO:teuthology.orchestra.run.smithi098.stdout:4905 write 0xb3ff770 thru 0xb408605 (0x8e96 bytes) 2024-09-16T13:18:21.783 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:21.783 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:21.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:21.795 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:21.792+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:21.797 INFO:teuthology.orchestra.run.smithi098.stdout:4906 trunc from 0xb408606 to 0x6b5ab81 2024-09-16T13:18:21.816 INFO:teuthology.orchestra.run.smithi098.stdout:4907 read 0x5105702 thru 0x510e7aa (0x90a9 bytes) 2024-09-16T13:18:21.817 INFO:teuthology.orchestra.run.smithi098.stdout:4908 read 0x4fad235 thru 0x4fb07c4 (0x3590 bytes) 2024-09-16T13:18:21.817 INFO:teuthology.orchestra.run.smithi098.stdout:4909 read 0x5507344 thru 0x55136e5 (0xc3a2 bytes) 2024-09-16T13:18:21.819 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:21.833 INFO:teuthology.orchestra.run.smithi098.stdout:4911 write 0x8dd4311 thru 0x8ddb835 (0x7525 bytes) 2024-09-16T13:18:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:4912 trunc from 0x8ddb836 to 0xc3433d0 2024-09-16T13:18:21.860 INFO:teuthology.orchestra.run.smithi098.stdout:4913 writesame 0xa89522 thru 0xa8fe06 data_size 0x1(0x68e5 bytes) 2024-09-16T13:18:21.890 INFO:teuthology.orchestra.run.smithi098.stdout:4915 read 0x4583b0f thru 0x459164e (0xdb40 bytes) 2024-09-16T13:18:21.898 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:21.891+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:21.902 INFO:teuthology.orchestra.run.smithi098.stdout:4916 trunc from 0xc3433d0 to 0x71706d4 2024-09-16T13:18:21.904 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:21.905 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:21.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:21.924 INFO:teuthology.orchestra.run.smithi098.stdout:4917 write 0xf28a7 thru 0xfeab5 (0xc20f bytes) 2024-09-16T13:18:21.928 INFO:teuthology.orchestra.run.smithi098.stdout:4918 punch from 0x6f5a66 to 0x6fdb5d, (0x80f7 bytes) 2024-09-16T13:18:21.928 INFO:teuthology.orchestra.run.smithi098.stdout:4919 punch from 0x592df3b to 0x5931cb5, (0x3d7a bytes) 2024-09-16T13:18:21.928 INFO:teuthology.orchestra.run.smithi098.stdout:4920 read 0x710396e thru 0x7112f6d (0xf600 bytes) 2024-09-16T13:18:21.943 INFO:teuthology.orchestra.run.smithi098.stdout:4921 write 0x9411704 thru 0x9413bf6 (0x24f3 bytes) 2024-09-16T13:18:21.948 INFO:teuthology.orchestra.run.smithi098.stdout:4922 punch from 0x816d809 to 0x8170901, (0x30f8 bytes) 2024-09-16T13:18:21.949 INFO:teuthology.orchestra.run.smithi098.stdout:4923 clone 36 order 20 su 65536 sc 8 2024-09-16T13:18:21.998 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:21.992+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:22.019 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:22.019 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:22.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:22.098 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:22.092+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:22.132 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:22.132 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:22.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:22.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:22.199 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:22.196+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:22.245 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:22.245 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:22.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:22.287 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:22.299 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:22.296+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:22.306 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:22.308 DEBUG:teuthology.orchestra.run.smithi151:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op apply-layout-settings --pool rbd 2024-09-16T13:18:22.325 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:22.350 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:22.358 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:22.358 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:22.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:22.368 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:22.370 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone35 from 0x9413bf7 (overlap 0x1c8039d) to 0xe401ce 2024-09-16T13:18:22.391 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:22.400 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:22.391+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:22.417 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:22.418 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:22.418 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:22.418 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:22.418 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:22.418 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:22.418 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:22.418 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:22.418 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:22.421 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 21.646089294999999, 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.474 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:18:22.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:18:22.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:18:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:18:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:18:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75103 3.1d 3:ba38d27a:::rbd_header.112d9326a5dd:head [watch reconnect cookie 140083112227968 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.434558+0000", 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.034002901, 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043398552999999999, 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75103 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.482 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434558+0000", 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434558+0000", 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434559+0000", 2024-09-16T13:18:22.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7399999999999991e-07 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434561+0000", 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4269999999999998e-06 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434567+0000", 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7370000000000003e-06 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.434570+0000", 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.495e-06 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.468996+0000", 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.034425509999999999 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.469084+0000", 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7750999999999994e-05 2024-09-16T13:18:22.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477956+0000", 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0088726589999999998 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75235 3.1d 3:babb8944:::rbd_data.112d3c2065c6.000000000000000b:25 [read 1441792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435173+0000", 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0333872799999999, 2024-09-16T13:18:22.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040680301000000002, 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75235 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435173+0000", 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435173+0000", 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435174+0000", 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435175+0000", 2024-09-16T13:18:22.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6750000000000001e-06 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435181+0000", 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0240000000000003e-06 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435186+0000", 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7310000000000001e-06 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.475667+0000", 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040481348 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.475761+0000", 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3919999999999995e-05 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.475854+0000", 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2215999999999999e-05 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.490 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75236 3.1d 3:babb8944:::rbd_data.112d3c2065c6.000000000000000b:25 [read 1769472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435192+0000", 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.033368501, 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040875799999999997, 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75236 2024-09-16T13:18:22.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435192+0000", 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435192+0000", 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435192+0000", 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8400000000000005e-07 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435194+0000", 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9259999999999999e-06 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435201+0000", 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5640000000000002e-06 2024-09-16T13:18:22.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435204+0000", 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1829999999999998e-06 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.475889+0000", 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040684413000000003 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.475986+0000", 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7050000000000001e-05 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476068+0000", 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2180000000000003e-05 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75237 3.1d 3:babb8944:::rbd_data.112d3c2065c6.000000000000000b:25 [read 2064384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.495 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435208+0000", 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0333522880000001, 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041996782000000003, 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75237 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435208+0000", 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435208+0000", 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435209+0000", 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435210+0000", 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8080000000000001e-06 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435216+0000", 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3539999999999999e-06 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435219+0000", 2024-09-16T13:18:22.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3419999999999999e-06 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477065+0000", 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041845787000000002 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477137+0000", 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2014000000000005e-05 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477205+0000", 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8089999999999994e-05 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75274 3.d 3:b0e641d3:::rbd_data.112d3c2065c6.0000000000000019:25 [read 196608~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435223+0000", 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0333379439999999, 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041113480000000001, 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.500 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75274 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435223+0000", 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435223+0000", 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435223+0000", 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9900000000000001e-07 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435225+0000", 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.751e-06 2024-09-16T13:18:22.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435230+0000", 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5779999999999998e-06 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435239+0000", 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9930000000000001e-06 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476098+0000", 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040858271000000002 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476188+0000", 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0575000000000006e-05 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476336+0000", 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147913 2024-09-16T13:18:22.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75275 3.d 3:b0e641d3:::rbd_data.112d3c2065c6.0000000000000019:25 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435253+0000", 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.033307502, 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041335887000000002, 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.505 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75275 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435253+0000", 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435253+0000", 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435254+0000", 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435261+0000", 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435267+0000", 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7320000000000001e-06 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435275+0000", 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5020000000000003e-06 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476359+0000", 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041084260999999997 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476442+0000", 2024-09-16T13:18:22.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3015000000000004e-05 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476589+0000", 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014677700000000001 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75276 3.d 3:b0e641d3:::rbd_data.112d3c2065c6.0000000000000019:25 [read 1146880~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435279+0000", 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0332816459999998, 2024-09-16T13:18:22.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041563015000000002, 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75276 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435279+0000", 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435279+0000", 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435279+0000", 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6600000000000002e-07 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435281+0000", 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435287+0000", 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7119999999999997e-06 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435290+0000", 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2440000000000002e-06 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476615+0000", 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041324913999999997 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476702+0000", 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7394999999999999e-05 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476842+0000", 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013951499999999999 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75277 3.d 3:b0e641d3:::rbd_data.112d3c2065c6.0000000000000019:25 [read 1441792~31893] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435294+0000", 2024-09-16T13:18:22.513 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0332667629999999, 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042150939999999998, 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75277 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435294+0000", 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435294+0000", 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435294+0000", 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7599999999999998e-07 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435296+0000", 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7400000000000001e-06 2024-09-16T13:18:22.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435303+0000", 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6789999999999999e-06 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435306+0000", 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.281e-06 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476884+0000", 2024-09-16T13:18:22.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041577792000000002 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.476950+0000", 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6031999999999994e-05 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477445+0000", 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049503999999999998 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75278 3.d 3:b0e641d3:::rbd_data.112d3c2065c6.0000000000000019:25 [read 1769472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435310+0000", 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0332505830000001, 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042650988000000001, 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75278 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435310+0000", 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435310+0000", 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435310+0000", 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435312+0000", 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.844e-06 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435318+0000", 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6949999999999997e-06 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435321+0000", 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.382e-06 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477501+0000", 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042179609 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477660+0000", 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015907700000000001 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477961+0000", 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030097499999999998 2024-09-16T13:18:22.521 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75313 3.1d 3:be4d3b50:::rbd_data.112d3c2065c6.0000000000000028:25 [read 196608~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435446+0000", 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.033114366, 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042256112999999998, 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.522 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75313 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435446+0000", 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435446+0000", 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435447+0000", 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435448+0000", 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7990000000000001e-06 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435453+0000", 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0490000000000003e-06 2024-09-16T13:18:22.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435457+0000", 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1190000000000001e-06 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477469+0000", 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042012723000000002 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477558+0000", 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8477e-05 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477702+0000", 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014455800000000001 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75314 3.1d 3:be4d3b50:::rbd_data.112d3c2065c6.0000000000000028:25 [read 819200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435461+0000", 2024-09-16T13:18:22.526 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0330998509999998, 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042415044999999998, 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75314 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435461+0000", 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435461+0000", 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435461+0000", 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8800000000000003e-07 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435463+0000", 2024-09-16T13:18:22.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9080000000000002e-06 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435469+0000", 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7660000000000004e-06 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435472+0000", 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1659999999999998e-06 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477725+0000", 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042253023000000001 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477801+0000", 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6370000000000002e-05 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.477876+0000", 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4324000000000002e-05 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75315 3.1d 3:be4d3b50:::rbd_data.112d3c2065c6.0000000000000028:25 [read 1146880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435476+0000", 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.033085008, 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043258913000000003, 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.531 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75315 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435476+0000", 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435476+0000", 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435476+0000", 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5499999999999998e-07 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435478+0000", 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7260000000000001e-06 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435483+0000", 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435488+0000", 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2230000000000001e-06 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478020+0000", 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042532055999999999 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478590+0000", 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057009699999999999 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478734+0000", 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014470800000000001 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75316 3.1d 3:be4d3b50:::rbd_data.112d3c2065c6.0000000000000028:25 [read 1441792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435491+0000", 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0330694540000001, 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043410124000000001, 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75316 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435491+0000", 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435491+0000", 2024-09-16T13:18:22.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435491+0000", 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8099999999999998e-07 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435493+0000", 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7e-06 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435499+0000", 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4110000000000004e-06 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435502+0000", 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4620000000000001e-06 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478762+0000", 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043260028999999998 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478822+0000", 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0204999999999997e-05 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478901+0000", 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8936000000000005e-05 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.539 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75317 3.1d 3:be4d3b50:::rbd_data.112d3c2065c6.0000000000000028:25 [read 1769472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435505+0000", 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0330550280000002, 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044500790999999998, 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75317 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435505+0000", 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435505+0000", 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435506+0000", 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8500000000000002e-07 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435507+0000", 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4839999999999999e-06 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435513+0000", 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2920000000000003e-06 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435516+0000", 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5439999999999999e-06 2024-09-16T13:18:22.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479903+0000", 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044387175000000001 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479955+0000", 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1721999999999997e-05 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.480006+0000", 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1088999999999998e-05 2024-09-16T13:18:22.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75382 3.d 3:b6152032:::rbd_data.112d3c2065c6.0000000000000048:25 [read 491520~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435520+0000", 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.033040422, 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043630454999999999, 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75382 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435520+0000", 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435520+0000", 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435520+0000", 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435522+0000", 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435528+0000", 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1880000000000001e-06 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435531+0000", 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999999e-06 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.478928+0000", 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043397133999999997 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479017+0000", 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8769000000000003e-05 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479151+0000", 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013375600000000001 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75383 3.d 3:b6152032:::rbd_data.112d3c2065c6.0000000000000048:25 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435540+0000", 2024-09-16T13:18:22.548 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0330203080000002, 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043809694000000003, 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75383 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435540+0000", 2024-09-16T13:18:22.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435540+0000", 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435541+0000", 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1800000000000001e-07 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435551+0000", 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0457e-05 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435559+0000", 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4669999999999998e-06 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435562+0000", 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479183+0000", 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043620402000000003 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479274+0000", 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1289e-05 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479350+0000", 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5862e-05 2024-09-16T13:18:22.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75384 3.d 3:b6152032:::rbd_data.112d3c2065c6.0000000000000048:25 [read 1114112~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435567+0000", 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0329937139999998, 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043967154000000001, 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.553 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75384 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435567+0000", 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435567+0000", 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435567+0000", 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435569+0000", 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9750000000000001e-06 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435575+0000", 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5659999999999997e-06 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435583+0000", 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.473e-06 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479377+0000", 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043793374000000003 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479433+0000", 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6422000000000003e-05 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479534+0000", 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100929 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75385 3.d 3:b6152032:::rbd_data.112d3c2065c6.0000000000000048:25 [read 1441792~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435588+0000", 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0329727950000001, 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044113678000000003, 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.557 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75385 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435588+0000", 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435588+0000", 2024-09-16T13:18:22.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435588+0000", 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2399999999999999e-07 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435590+0000", 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.119e-06 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435598+0000", 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8949999999999998e-06 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435604+0000", 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7760000000000003e-06 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479557+0000", 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043952705000000002 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479606+0000", 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9305999999999999e-05 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479701+0000", 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5452999999999996e-05 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75386 3.d 3:b6152032:::rbd_data.112d3c2065c6.0000000000000048:25 [read 2064384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.435608+0000", 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.0329524600000002, 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044265631, 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75386 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.562 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435608+0000", 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435608+0000", 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435608+0000", 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3599999999999999e-07 2024-09-16T13:18:22.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435611+0000", 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1689999999999999e-06 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435618+0000", 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7270000000000007e-06 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.435628+0000", 2024-09-16T13:18:22.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1880000000000008e-06 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479724+0000", 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044096718999999999 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479770+0000", 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5274000000000001e-05 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.479874+0000", 2024-09-16T13:18:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010411799999999999 2024-09-16T13:18:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.566 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.566 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:22.567 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #34, image image_client.0-clone34 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent35 2024-09-16T13:18:22.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:22.568 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:22.559+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:22.668 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:22.659+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:22.724 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:22.724 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:22.724 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:22.724 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:22.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:22.769 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:22.764+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:22.830 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:22.830 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:22.830 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:22.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:22.870 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:22.860+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:22.940 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:22.940 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:22.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:22.940 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.053169664999999, 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.941 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:18:22.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:18:22.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.945 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.05315083, 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:18:22.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:22.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:18:22.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:18:22.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.950 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.053131691000001, 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.951 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:18:22.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.955 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.956 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.956 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.956 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:22.956 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.053114633, 2024-09-16T13:18:22.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:18:22.956 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.956 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.956 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:18:22.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:18:22.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:22.961 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.053097272999999, 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:18:22.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:18:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:18:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:18:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:18:22.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.966 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.053079564000001, 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:18:22.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:18:22.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.971 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.053062265000001, 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:18:22.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.053027167, 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.053004011999999, 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.981 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:18:22.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:18:22.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:18:22.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:18:22.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:18:22.985 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.985 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.986 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.986 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.986 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:22.986 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.052962993000001, 2024-09-16T13:18:22.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:18:22.986 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:22.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:22.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:18:22.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:18:22.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:18:22.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:18:22.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:18:22.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:18:22.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:18:22.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:18:22.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:22.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.052940093, 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:18:22.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:22.993 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:22.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:22.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:22.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:22.993 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:18:22.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.993 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:22.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:22.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:22.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:22.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:22.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:22.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:18:22.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:22.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:23.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:18:23.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:18:23.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:23.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:18:23.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:18:23.000 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.000 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.052915043999999, 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:18:23.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:18:23.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:18:23.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.052889823000001, 2024-09-16T13:18:23.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:18:23.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.052871457999998, 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:23.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:23.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:18:23.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:18:23.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.052608321000001, 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:23.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:23.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.016 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:18:23.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.016 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:23.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:23.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:23.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:18:23.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:23.020 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.052583313, 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:18:23.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:18:23.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:23.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:18:23.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:23.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:23.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:18:23.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:18:23.025 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.025 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.026 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.026 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:23.026 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:23.026 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.052554981, 2024-09-16T13:18:23.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:18:23.026 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:23.026 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:23.026 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:23.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:18:23.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:18:23.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.052266283000002, 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:18:23.031 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:18:23.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:23.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.052246173, 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:18:23.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:23.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:18:23.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.048045249000001, 2024-09-16T13:18:23.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:18:23.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:23.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:18:23.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:18:23.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.045 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:23.045 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:23.045 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:23.047 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:23.047 INFO:teuthology.orchestra.run.smithi098.stdout:4925 trunc from 0x9413bf7 to 0x27cac09 2024-09-16T13:18:23.047 INFO:teuthology.orchestra.run.smithi098.stdout:4926 trunc from 0x27cac09 to 0xacfc4c3 2024-09-16T13:18:23.047 INFO:teuthology.orchestra.run.smithi098.stdout:4928 write 0xe25ed6a thru 0xe2614b2 (0x2749 bytes) 2024-09-16T13:18:23.048 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:23.039+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:23.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:23.052 INFO:teuthology.orchestra.run.smithi098.stdout:4930 read 0xdff4b3c thru 0xe0042d8 (0xf79d bytes) 2024-09-16T13:18:23.052 INFO:teuthology.orchestra.run.smithi098.stdout:4931 write 0xf4fb3b thru 0xf50c68 (0x112e bytes) 2024-09-16T13:18:23.076 INFO:teuthology.orchestra.run.smithi098.stdout:4933 writesame 0x7dcbb55 thru 0x7dcfc9c data_size 0x4(0x4148 bytes) 2024-09-16T13:18:23.083 INFO:teuthology.orchestra.run.smithi098.stdout:4935 write 0x53ce6c7 thru 0x53dbc56 (0xd590 bytes) 2024-09-16T13:18:23.091 INFO:teuthology.orchestra.run.smithi098.stdout:4938 write 0x3311ec8 thru 0x331a91f (0x8a58 bytes) 2024-09-16T13:18:23.103 INFO:teuthology.orchestra.run.smithi098.stdout:4940 trunc from 0xe2614b3 to 0xaf43932 2024-09-16T13:18:23.135 INFO:teuthology.orchestra.run.smithi098.stdout:4944 write 0xb78e1fd thru 0xb797b27 (0x992b bytes) 2024-09-16T13:18:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:4945 writesame 0x68bf06a thru 0x68bfa64 data_size 0x7(0x9fb bytes) 2024-09-16T13:18:23.147 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:23.140+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:4947 writesame 0x663b3fe thru 0x663c8e5 data_size 0xc(0x14e8 bytes) 2024-09-16T13:18:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:23.151 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:23.151 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:23.156 INFO:teuthology.orchestra.run.smithi098.stdout:4948 punch from 0x70bd420 to 0x70c9676, (0xc256 bytes) 2024-09-16T13:18:23.156 INFO:teuthology.orchestra.run.smithi098.stdout:4949 write 0x6ea963d thru 0x6eb8525 (0xeee9 bytes) 2024-09-16T13:18:23.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:23.162 INFO:teuthology.orchestra.run.smithi098.stdout:4951 read 0x2259055 thru 0x22605a7 (0x7553 bytes) 2024-09-16T13:18:23.162 INFO:teuthology.orchestra.run.smithi098.stdout:4952 writesame 0x1e6b639 thru 0x1e700f9 data_size 0x1(0x4ac1 bytes) 2024-09-16T13:18:23.190 INFO:teuthology.orchestra.run.smithi098.stdout:4953 write 0x4fa8d2d thru 0x4fb5cb0 (0xcf84 bytes) 2024-09-16T13:18:23.210 INFO:teuthology.orchestra.run.smithi098.stdout:4954 write 0xd97de0a thru 0xd985c79 (0x7e70 bytes) 2024-09-16T13:18:23.219 INFO:teuthology.orchestra.run.smithi098.stdout:4955 write 0x5e7b844 thru 0x5e87aa9 (0xc266 bytes) 2024-09-16T13:18:23.228 INFO:teuthology.orchestra.run.smithi098.stdout:4956 punch from 0xac207ed to 0xac21ad0, (0x12e3 bytes) 2024-09-16T13:18:23.228 INFO:teuthology.orchestra.run.smithi098.stdout:4958 writesame 0x3806ff2 thru 0x380953f data_size 0x19(0x254e bytes) 2024-09-16T13:18:23.248 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:23.240+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:23.250 INFO:teuthology.orchestra.run.smithi098.stdout:4960 trunc from 0xd985c7a to 0x4618034 2024-09-16T13:18:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:23.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:18:23.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:4961 writesame 0xc0ff80f thru 0xc10a228 data_size 0x1(0xaa1a bytes) 2024-09-16T13:18:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:4963 read 0x9a42fc8 thru 0x9a432e0 (0x319 bytes) 2024-09-16T13:18:23.303 INFO:teuthology.orchestra.run.smithi098.stdout:4964 read 0x1d80c7d thru 0x1d8bcf4 (0xb078 bytes) 2024-09-16T13:18:23.304 INFO:teuthology.orchestra.run.smithi098.stdout:4966 writesame 0x314627b thru 0x31527df data_size 0x1(0xc565 bytes) 2024-09-16T13:18:23.349 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:23.340+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:23.350 INFO:teuthology.orchestra.run.smithi098.stdout:4967 read 0xc0114b thru 0xc09785 (0x863b bytes) 2024-09-16T13:18:23.352 INFO:teuthology.orchestra.run.smithi098.stdout:4968 read 0x68349af thru 0x683ea2a (0xa07c bytes) 2024-09-16T13:18:23.352 INFO:teuthology.orchestra.run.smithi098.stdout:4969 write 0x57ead7b thru 0x57f192d (0x6bb3 bytes) 2024-09-16T13:18:23.362 INFO:teuthology.orchestra.run.smithi098.stdout:4970 trunc from 0xc10a229 to 0xb48833e 2024-09-16T13:18:23.379 INFO:teuthology.orchestra.run.smithi098.stdout:4972 write 0x1f66ad thru 0x20444f (0xdda3 bytes) 2024-09-16T13:18:23.402 INFO:teuthology.orchestra.run.smithi098.stdout:4973 write 0x2f59c97 thru 0x2f64f71 (0xb2db bytes) 2024-09-16T13:18:23.413 INFO:teuthology.orchestra.run.smithi098.stdout:4974 trunc from 0xb48833e to 0xa87adad 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66873 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4034560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8009284260000005, 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034635435999999999, 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.421 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66873 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615785+0000", 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615786+0000", 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615792+0000", 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1900000000000003e-06 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615795+0000", 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9450000000000002e-06 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650232+0000", 2024-09-16T13:18:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034437669999999997 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650301+0000", 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8329000000000003e-05 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650420+0000", 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119132 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66874 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4042752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8008952199999992, 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034827871000000003, 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66874 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.426 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615820+0000", 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.063e-06 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615826+0000", 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9739999999999999e-06 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615835+0000", 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5920000000000001e-06 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650445+0000", 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034610100999999997 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650512+0000", 2024-09-16T13:18:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650645+0000", 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013316399999999999 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66875 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4050944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8008743109999994, 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032338000000003, 2024-09-16T13:18:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66875 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615839+0000", 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615841+0000", 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9250000000000002e-06 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615846+0000", 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.711e-06 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615851+0000", 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9579999999999998e-06 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650676+0000", 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034824524000000003 2024-09-16T13:18:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650758+0000", 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1547999999999999e-05 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650871+0000", 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113244 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66876 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4059136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8008574159999995, 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035239591000000001, 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66876 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615856+0000", 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615858+0000", 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.857e-06 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615863+0000", 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5389999999999998e-06 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615866+0000", 2024-09-16T13:18:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650899+0000", 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032448000000001 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650973+0000", 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3627000000000005e-05 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651095+0000", 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012239400000000001 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66877 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4067328~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.800842716, 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035445691000000001, 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66877 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615872+0000", 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.703e-06 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615877+0000", 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2650000000000004e-06 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615881+0000", 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.292e-06 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651123+0000", 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035242373 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651198+0000", 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5096000000000001e-05 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651316+0000", 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011757 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66878 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4075520~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8008288229999998, 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035647261999999999, 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66878 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5400000000000002e-07 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615890+0000", 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6189999999999999e-06 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615896+0000", 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6529999999999999e-06 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615900+0000", 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0069999999999999e-06 2024-09-16T13:18:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651337+0000", 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035437877999999999 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651395+0000", 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7219999999999998e-05 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651531+0000", 2024-09-16T13:18:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013653100000000001 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66879 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.800809482, 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035838241999999999, 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66879 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615904+0000", 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615906+0000", 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615911+0000", 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6679999999999998e-06 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615914+0000", 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.151e-06 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651558+0000", 2024-09-16T13:18:23.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035643672000000001 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651624+0000", 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6070999999999998e-05 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651742+0000", 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011729499999999999 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66880 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4091904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8007949019999998, 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036042148000000003, 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66880 2024-09-16T13:18:23.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2099999999999997e-07 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615924+0000", 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615930+0000", 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6250000000000004e-06 2024-09-16T13:18:23.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615933+0000", 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3950000000000001e-06 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651773+0000", 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035839579000000003 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651846+0000", 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2602999999999996e-05 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651960+0000", 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114319 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.457 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66881 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4100096~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8007738849999999, 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036242563999999998, 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66881 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:23.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615941+0000", 2024-09-16T13:18:23.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.72e-06 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615946+0000", 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.259e-06 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615950+0000", 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651985+0000", 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036035821000000003 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652058+0000", 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2651999999999995e-05 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652181+0000", 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123428 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66882 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4108288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8007598550000008, 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036437628999999999, 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66882 2024-09-16T13:18:23.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7300000000000002e-07 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615955+0000", 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8199999999999999e-06 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615961+0000", 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2349999999999996e-06 2024-09-16T13:18:23.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615965+0000", 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4620000000000001e-06 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652207+0000", 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036241873000000001 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652276+0000", 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9105999999999999e-05 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652391+0000", 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011476000000000001 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66883 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4116480~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:23.466 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8007446320000007, 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036642620000000001, 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66883 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.77e-07 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615970+0000", 2024-09-16T13:18:23.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7519999999999999e-06 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615977+0000", 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9349999999999997e-06 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615985+0000", 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1389999999999995e-06 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652416+0000", 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036430381999999997 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652487+0000", 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1698000000000004e-05 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652611+0000", 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123337 2024-09-16T13:18:23.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66884 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4124672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8004802049999995, 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036613363000000003, 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.471 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66884 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.16e-07 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616237+0000", 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5719999999999999e-06 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616245+0000", 2024-09-16T13:18:23.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8820000000000001e-06 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616251+0000", 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9630000000000003e-06 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652638+0000", 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036387705999999999 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652719+0000", 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0469999999999994e-05 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652846+0000", 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126954 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66885 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4132864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.475 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8004576530000005, 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036785929000000002, 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66885 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:23.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0999999999999999e-07 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616258+0000", 2024-09-16T13:18:23.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0269999999999998e-06 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616263+0000", 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8760000000000002e-06 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616267+0000", 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652873+0000", 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036606376000000003 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652956+0000", 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2824000000000005e-05 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653041+0000", 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4918000000000002e-05 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66886 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4141056~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8004412240000001, 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078279999999998, 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66886 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:23.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616274+0000", 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.73e-06 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616279+0000", 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616283+0000", 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1420000000000002e-06 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653119+0000", 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036836016999999999 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653204+0000", 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5575999999999995e-05 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653350+0000", 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014569499999999999 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66887 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4149248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8004267210000009, 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037308557999999999, 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.484 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66887 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:23.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.84e-07 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616288+0000", 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7099999999999999e-06 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616294+0000", 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4029999999999998e-06 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616297+0000", 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0079999999999998e-06 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653375+0000", 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078883 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653455+0000", 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0034e-05 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653595+0000", 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013913599999999999 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66888 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4157440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8004123500000002, 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537323999999997, 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66888 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616301+0000", 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616302+0000", 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616308+0000", 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616317+0000", 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2229999999999996e-06 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653620+0000", 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037302572999999999 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653701+0000", 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1771e-05 2024-09-16T13:18:23.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653838+0000", 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013640599999999999 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66889 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4165632~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8003918219999999, 2024-09-16T13:18:23.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037755019000000001, 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66889 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:23.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2300000000000002e-07 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616323+0000", 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.905e-06 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616329+0000", 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4639999999999997e-06 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616332+0000", 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0180000000000002e-06 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653862+0000", 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037529973000000001 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653937+0000", 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4954000000000001e-05 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654076+0000", 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013928200000000001 2024-09-16T13:18:23.497 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66890 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4173824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8003764160000006, 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037917073000000003, 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66890 2024-09-16T13:18:23.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.499 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:23.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616337+0000", 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616338+0000", 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616371+0000", 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2806999999999999e-05 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616375+0000", 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.827e-06 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654115+0000", 2024-09-16T13:18:23.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.03773986 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654181+0000", 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5652000000000001e-05 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654253+0000", 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2789999999999999e-05 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66891 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4182016~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8003285449999993, 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038160044999999997, 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66891 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616385+0000", 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:18:23.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616387+0000", 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.356e-06 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616393+0000", 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9519999999999999e-06 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616396+0000", 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2959999999999999e-06 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654302+0000", 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037905031999999998 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654395+0000", 2024-09-16T13:18:23.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3560000000000005e-05 2024-09-16T13:18:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654544+0000", 2024-09-16T13:18:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014916500000000001 2024-09-16T13:18:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66892 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4190208~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.8003127340000002, 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038342319999999999, 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66892 2024-09-16T13:18:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616407+0000", 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4060000000000002e-06 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616412+0000", 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-06 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616419+0000", 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.037e-06 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654571+0000", 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038152835000000003 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654625+0000", 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3643000000000001e-05 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654742+0000", 2024-09-16T13:18:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117348 2024-09-16T13:18:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.512 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:23.513 INFO:teuthology.orchestra.run.smithi098.stdout:4975 write 0x189668 thru 0x18edc0 (0x5759 bytes) 2024-09-16T13:18:23.513 INFO:teuthology.orchestra.run.smithi151.stderr:Nothing to do for non-filestore backend 2024-09-16T13:18:23.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4976 trunc from 0xa87adad to 0x58e1ac0 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4979 writesame 0x7d1e465 thru 0x7d207b3 data_size 0x17(0x234f bytes) 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4981 trunc from 0x7d207b4 to 0xc4222fb 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4982 write 0x55ae814 thru 0x55b57e8 (0x6fd5 bytes) 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4983 read 0xa821819 thru 0xa82f819 (0xe001 bytes) 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4984 read 0x425419a thru 0x4258b58 (0x49bf bytes) 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4985 writesame 0xcf48afd thru 0xcf53086 data_size 0x15(0xa58a bytes) 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4986 write 0x224a13e thru 0x224e5f6 (0x44b9 bytes) 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4990 read 0x6ec40f5 thru 0x6ed2660 (0xe56c bytes) 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4991 write 0x9103298 thru 0x910bad4 (0x883d bytes) 2024-09-16T13:18:23.516 INFO:teuthology.orchestra.run.smithi098.stdout:4995 write 0x784ca66 thru 0x7856526 (0x9ac1 bytes) 2024-09-16T13:18:23.517 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:23.507+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:23.519 INFO:teuthology.orchestra.run.smithi098.stdout:4996 write 0x7d5de21 thru 0x7d5f536 (0x1716 bytes) 2024-09-16T13:18:23.522 INFO:teuthology.orchestra.run.smithi098.stdout:4998 writesame 0x25ae376 thru 0x25af77d data_size 0x4(0x1408 bytes) 2024-09-16T13:18:23.535 INFO:teuthology.orchestra.run.smithi098.stdout:4999 read 0x45e6bf5 thru 0x45ea8a2 (0x3cae bytes) 2024-09-16T13:18:23.535 INFO:teuthology.orchestra.run.smithi098.stdout:5000 write 0x861a236 thru 0x8629373 (0xf13e bytes) 2024-09-16T13:18:23.542 INFO:teuthology.orchestra.run.smithi098.stdout:5001 read 0x3daf17e thru 0x3db3bd9 (0x4a5c bytes) 2024-09-16T13:18:23.542 INFO:teuthology.orchestra.run.smithi098.stdout:5002 write 0x6321444 thru 0x6324775 (0x3332 bytes) 2024-09-16T13:18:23.547 INFO:teuthology.orchestra.run.smithi098.stdout:5004 read 0x5aae7b6 thru 0x5ab52ea (0x6b35 bytes) 2024-09-16T13:18:23.548 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:23.548 INFO:teuthology.orchestra.run.smithi098.stdout:5007 writesame 0x9d93a7d thru 0x9d978ba data_size 0x1f(0x3e3e bytes) 2024-09-16T13:18:23.553 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:23.553 INFO:teuthology.orchestra.run.smithi098.stdout:5009 read 0x1199b09 thru 0x11a6073 (0xc56b bytes) 2024-09-16T13:18:23.569 INFO:teuthology.orchestra.run.smithi098.stdout:5010 trunc from 0xcf53087 to 0x3b3ac05 2024-09-16T13:18:23.593 INFO:teuthology.orchestra.run.smithi098.stdout:5011 trunc from 0x3b3ac05 to 0x190c7fa 2024-09-16T13:18:23.615 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:23.608+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:23.628 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:82229 3.19 3:9b519492:::rbd_object_map.112d12abb20f:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:23.624116+0000", 2024-09-16T13:18:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0018423339999999999, 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0018618160000000001, 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for sub ops", 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 82229 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.624116+0000", 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.624116+0000", 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.624117+0000", 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6400000000000001e-07 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.624121+0000", 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.544e-06 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.624132+0000", 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1066e-05 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.624136+0000", 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3510000000000004e-06 2024-09-16T13:18:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.624189+0000", 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2611000000000002e-05 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.624291+0000", 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00010222599999999999 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.624696+0000", 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000404678 2024-09-16T13:18:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.625601+0000", 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00090517699999999996 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 1 2024-09-16T13:18:23.633 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:23.637 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:23.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:23.657 INFO:teuthology.orchestra.run.smithi098.stdout:5015 write 0x809795d thru 0x809c05e (0x4702 bytes) 2024-09-16T13:18:23.685 INFO:teuthology.orchestra.run.smithi098.stdout:5016 writesame 0xe351f28 thru 0xe35443b data_size 0xe(0x2514 bytes) 2024-09-16T13:18:23.685 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:23.686 INFO:teuthology.orchestra.run.smithi098.stdout:5019 punch from 0x2bb24f9 to 0x2bbe90e, (0xc415 bytes) 2024-09-16T13:18:23.686 INFO:teuthology.orchestra.run.smithi098.stdout:5020 write 0x6741ac3 thru 0x67469f9 (0x4f37 bytes) 2024-09-16T13:18:23.692 INFO:teuthology.orchestra.run.smithi098.stdout:5021 punch from 0x8361623 to 0x836e69f, (0xd07c bytes) 2024-09-16T13:18:23.712 INFO:teuthology.orchestra.run.smithi098.stdout:5022 trunc from 0xe35443c to 0x1f96840 2024-09-16T13:18:23.715 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:23.708+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:23.758 INFO:teuthology.orchestra.run.smithi098.stdout:5023 write 0x4fa4c76 thru 0x4faf474 (0xa7ff bytes) 2024-09-16T13:18:23.765 INFO:teuthology.orchestra.run.smithi098.stdout:5024 punch from 0x48477b0 to 0x484bae3, (0x4333 bytes) 2024-09-16T13:18:23.765 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:23.768 INFO:teuthology.orchestra.run.smithi098.stdout:5027 trunc from 0x4faf475 to 0x361f662 2024-09-16T13:18:23.772 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:23.772 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:23.772 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:23.772 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:23.772 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:23.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:23.788 INFO:teuthology.orchestra.run.smithi098.stdout:5028 read 0x2273703 thru 0x227b427 (0x7d25 bytes) 2024-09-16T13:18:23.790 INFO:teuthology.orchestra.run.smithi098.stdout:5029 trunc from 0x361f662 to 0x4171fdb 2024-09-16T13:18:23.802 INFO:teuthology.orchestra.run.smithi098.stdout:5031 read 0x12000aa thru 0x1202b1d (0x2a74 bytes) 2024-09-16T13:18:23.816 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:23.807+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:23.821 INFO:teuthology.orchestra.run.smithi098.stdout:5032 write 0x6979104 thru 0x697ec9a (0x5b97 bytes) 2024-09-16T13:18:23.828 INFO:teuthology.orchestra.run.smithi098.stdout:5034 read 0x2c04344 thru 0x2c0d389 (0x9046 bytes) 2024-09-16T13:18:23.828 INFO:teuthology.orchestra.run.smithi098.stdout:5035 write 0x642c579 thru 0x642ffd4 (0x3a5c bytes) 2024-09-16T13:18:23.834 INFO:teuthology.orchestra.run.smithi098.stdout:5036 punch from 0x25d7588 to 0x25e1634, (0xa0ac bytes) 2024-09-16T13:18:23.844 INFO:teuthology.orchestra.run.smithi098.stdout:5037 trunc from 0x697ec9b to 0x13fda03 2024-09-16T13:18:23.900 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64860 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3899392~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332928513000001, 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035719773000000003, 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64860 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563336+0000", 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563342+0000", 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3240000000000002e-06 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563345+0000", 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:18:23.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.598973+0000", 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035627631999999999 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599006+0000", 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3495999999999999e-05 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599054+0000", 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7460999999999998e-05 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64861 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3907584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332913632, 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035815461, 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.905 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64861 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563351+0000", 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.854e-06 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563356+0000", 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5940000000000001e-06 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563360+0000", 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4060000000000001e-06 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599076+0000", 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035716171999999997 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599116+0000", 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9777999999999998e-05 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599164+0000", 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8307999999999999e-05 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64862 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332899039000001, 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035954097999999997, 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64862 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563364+0000", 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:23.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563365+0000", 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.821e-06 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563371+0000", 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0769999999999998e-06 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563374+0000", 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3560000000000001e-06 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599187+0000", 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035813153 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599252+0000", 2024-09-16T13:18:23.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4873e-05 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599317+0000", 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5438000000000006e-05 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64863 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3923968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:23.914 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332873485, 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036051842000000001, 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64863 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:23.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563392+0000", 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3140000000000002e-06 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563398+0000", 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.066e-06 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563402+0000", 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599339+0000", 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035937678000000001 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599377+0000", 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599441+0000", 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3380999999999995e-05 2024-09-16T13:18:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64864 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3932160~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332857081, 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036187015000000003, 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64864 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563406+0000", 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563411+0000", 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563417+0000", 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-16T13:18:23.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563421+0000", 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599458+0000", 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036036870999999998 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599515+0000", 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7067000000000001e-05 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599592+0000", 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7457000000000001e-05 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64865 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3940352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332837691, 2024-09-16T13:18:23.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036265720000000001, 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64865 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563427+0000", 2024-09-16T13:18:23.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.001e-06 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563433+0000", 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5489999999999997e-06 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563436+0000", 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.484e-06 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599613+0000", 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036176815000000001 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599646+0000", 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3192999999999999e-05 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599690+0000", 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4240000000000003e-05 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64866 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3948544~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332821913, 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036355749999999999, 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64866 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563441+0000", 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:23.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563443+0000", 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0870000000000001e-06 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563448+0000", 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.062e-06 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563451+0000", 2024-09-16T13:18:23.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599712+0000", 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036261007999999997 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599753+0000", 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0476999999999999e-05 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599796+0000", 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3226000000000003e-05 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64867 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3956736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332807173000001, 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036484356000000003, 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64867 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563456+0000", 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563458+0000", 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9980000000000002e-06 2024-09-16T13:18:23.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563463+0000", 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563466+0000", 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599810+0000", 2024-09-16T13:18:23.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036343683000000002 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599864+0000", 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3565e-05 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599939+0000", 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5889999999999993e-05 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64868 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3964928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332792015000001, 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036573319999999999, 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64868 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563471+0000", 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563473+0000", 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563479+0000", 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.178e-06 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563483+0000", 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6339999999999999e-06 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599963+0000", 2024-09-16T13:18:23.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036480090999999999 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600001+0000", 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8340999999999998e-05 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600044+0000", 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.941 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64869 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3973120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332776041000001, 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036707818000000003, 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64869 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563487+0000", 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9100000000000003e-07 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563489+0000", 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563494+0000", 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563498+0000", 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.174e-06 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600061+0000", 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036563127000000001 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600119+0000", 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8476000000000001e-05 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600194+0000", 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4869000000000004e-05 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64870 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3981312~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.946 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332760999, 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036852528000000002, 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64870 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:23.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563502+0000", 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563504+0000", 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9479999999999998e-06 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563509+0000", 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563512+0000", 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3239999999999999e-06 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600218+0000", 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036705812999999997 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600280+0000", 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2298000000000006e-05 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600354+0000", 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3343000000000006e-05 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64871 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3989504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332746293, 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036938311000000001, 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64871 2024-09-16T13:18:23.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563518+0000", 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9130000000000001e-06 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563523+0000", 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563529+0000", 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0710000000000002e-06 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036847958 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600411+0000", 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4388999999999999e-05 2024-09-16T13:18:23.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600454+0000", 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3411000000000002e-05 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64872 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3997696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332729786, 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037077038999999999, 2024-09-16T13:18:23.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64872 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563533+0000", 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563535+0000", 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1270000000000001e-06 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563540+0000", 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3120000000000001e-06 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563544+0000", 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4699999999999998e-06 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600476+0000", 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036931691000000003 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600543+0000", 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7475e-05 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600610+0000", 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6397999999999997e-05 2024-09-16T13:18:23.959 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64873 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4005888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332714761, 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037239282999999998, 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64873 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563550+0000", 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563555+0000", 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563558+0000", 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2009999999999999e-06 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600630+0000", 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037071933000000001 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600702+0000", 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1245e-05 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600787+0000", 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5290000000000005e-05 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64874 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4014080~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332700315, 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037394846000000002, 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64874 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0400000000000002e-07 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563565+0000", 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1900000000000002e-06 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563570+0000", 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4199999999999998e-06 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563574+0000", 2024-09-16T13:18:23.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600812+0000", 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037237600000000003 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4690000000000006e-05 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600957+0000", 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0463999999999994e-05 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64875 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4022272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332684520000001, 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537883000000001, 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64875 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.65e-07 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563580+0000", 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563587+0000", 2024-09-16T13:18:23.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2589999999999997e-06 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563591+0000", 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600982+0000", 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037390117 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601044+0000", 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2788999999999998e-05 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601116+0000", 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1282e-05 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.973 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64876 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4030464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332667195000001, 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037682571999999998, 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64876 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:23.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563596+0000", 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563613+0000", 2024-09-16T13:18:23.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7935999999999999e-05 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563620+0000", 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4740000000000002e-06 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563624+0000", 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601141+0000", 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037517170000000002 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601203+0000", 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2187999999999995e-05 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601278+0000", 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:23.977 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64877 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4038656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332633586, 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037813404000000002, 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.978 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64877 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563631+0000", 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1330000000000002e-06 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563637+0000", 2024-09-16T13:18:23.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563640+0000", 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601304+0000", 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037664438000000001 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601372+0000", 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7618000000000002e-05 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601442+0000", 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-05 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64878 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4046848~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.982 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332604413, 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037956456, 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64878 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563659+0000", 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563661+0000", 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-06 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563667+0000", 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3470000000000004e-06 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563671+0000", 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7170000000000002e-06 2024-09-16T13:18:23.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601473+0000", 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037801955999999998 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601537+0000", 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3960000000000004e-05 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601614+0000", 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64879 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4055040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.332587774, 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038485017000000003, 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64879 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4099999999999998e-07 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563677+0000", 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1299999999999999e-06 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563682+0000", 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563688+0000", 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3329999999999996e-06 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601640+0000", 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037951864000000002 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601705+0000", 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5573000000000004e-05 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.602160+0000", 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000454411 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:23.991 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:23.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:23.994 INFO:teuthology.orchestra.run.smithi098.stdout:5040 writesame 0x37cd571 thru 0x37d1281 data_size 0x1b(0x3d11 bytes) 2024-09-16T13:18:23.994 INFO:teuthology.orchestra.run.smithi098.stdout:5042 read 0x28cbe08 thru 0x28d5a90 (0x9c89 bytes) 2024-09-16T13:18:23.994 INFO:teuthology.orchestra.run.smithi098.stdout:5043 writesame 0x2a7718e thru 0x2a7f685 data_size 0xa(0x84f8 bytes) 2024-09-16T13:18:23.994 INFO:teuthology.orchestra.run.smithi098.stdout:5044 write 0xc875869 thru 0xc87b958 (0x60f0 bytes) 2024-09-16T13:18:23.994 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:23.984+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:23.995 INFO:teuthology.orchestra.run.smithi098.stdout:5045 write 0x16580ea thru 0x1665b24 (0xda3b bytes) 2024-09-16T13:18:23.998 INFO:teuthology.orchestra.run.smithi098.stdout:5046 read 0x635f429 thru 0x636cac6 (0xd69e bytes) 2024-09-16T13:18:23.999 INFO:teuthology.orchestra.run.smithi098.stdout:5047 read 0x4476971 thru 0x44864da (0xfb6a bytes) 2024-09-16T13:18:23.999 INFO:teuthology.orchestra.run.smithi098.stdout:5048 writesame 0x84d59c1 thru 0x84d8423 data_size 0x1(0x2a63 bytes) 2024-09-16T13:18:24.018 INFO:teuthology.orchestra.run.smithi098.stdout:5049 punch from 0xd8948b to 0xd8f221, (0x5d96 bytes) 2024-09-16T13:18:24.018 INFO:teuthology.orchestra.run.smithi098.stdout:5051 write 0x9c8b08a thru 0x9c958b7 (0xa82e bytes) 2024-09-16T13:18:24.020 DEBUG:teuthology.orchestra.run.smithi151:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op apply-layout-settings --pool pool_client.0 2024-09-16T13:18:24.026 INFO:teuthology.orchestra.run.smithi098.stdout:5052 read 0x8c5ec8e thru 0x8c6b488 (0xc7fb bytes) 2024-09-16T13:18:24.026 INFO:teuthology.orchestra.run.smithi098.stdout:5053 write 0x234ff26 thru 0x235995a (0x9a35 bytes) 2024-09-16T13:18:24.033 INFO:teuthology.orchestra.run.smithi098.stdout:5054 write 0x24e6af2 thru 0x24f4a53 (0xdf62 bytes) 2024-09-16T13:18:24.040 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.040 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.040 INFO:teuthology.orchestra.run.smithi098.stdout:5057 read 0x2bff29f thru 0x2c0c965 (0xd6c7 bytes) 2024-09-16T13:18:24.041 INFO:teuthology.orchestra.run.smithi098.stdout:5058 punch from 0xadf266a to 0xadff796, (0xd12c bytes) 2024-09-16T13:18:24.041 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.053 INFO:teuthology.orchestra.run.smithi098.stdout:5062 write 0xd8aefcf thru 0xd8b66b8 (0x76ea bytes) 2024-09-16T13:18:24.060 INFO:teuthology.orchestra.run.smithi098.stdout:5063 writesame 0xcf00f9c thru 0xcf02b39 data_size 0x23(0x1b9e bytes) 2024-09-16T13:18:24.066 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.067 INFO:teuthology.orchestra.run.smithi098.stdout:5066 writesame 0x6a7386e thru 0x6a739ce data_size 0x1(0x161 bytes) 2024-09-16T13:18:24.086 INFO:teuthology.orchestra.run.smithi098.stdout:5067 write 0xead1a6a thru 0xead8f02 (0x7499 bytes) 2024-09-16T13:18:24.093 INFO:teuthology.orchestra.run.smithi098.stdout:5068 write 0x3dab62b thru 0x3dae5f5 (0x2fcb bytes) 2024-09-16T13:18:24.093 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:24.087+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:24.099 INFO:teuthology.orchestra.run.smithi098.stdout:5069 punch from 0xad04606 to 0xad0924f, (0x4c49 bytes) 2024-09-16T13:18:24.099 INFO:teuthology.orchestra.run.smithi098.stdout:5070 read 0x2a6e61c thru 0x2a751b8 (0x6b9d bytes) 2024-09-16T13:18:24.099 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.099 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.100 INFO:teuthology.orchestra.run.smithi098.stdout:5074 read 0x466a37b thru 0x467a100 (0xfd86 bytes) 2024-09-16T13:18:24.100 INFO:teuthology.orchestra.run.smithi098.stdout:5076 write 0x92a8770 thru 0x92af183 (0x6a14 bytes) 2024-09-16T13:18:24.106 INFO:teuthology.orchestra.run.smithi098.stdout:5077 write 0xc384b78 thru 0xc39098c (0xbe15 bytes) 2024-09-16T13:18:24.112 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:24.112 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:24.112 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:24.112 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:24.114 INFO:teuthology.orchestra.run.smithi098.stdout:5078 read 0x7c9d0fe thru 0x7ca1ff8 (0x4efb bytes) 2024-09-16T13:18:24.114 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.114 INFO:teuthology.orchestra.run.smithi098.stdout:5081 punch from 0x2b18418 to 0x2b2211d, (0x9d05 bytes) 2024-09-16T13:18:24.115 INFO:teuthology.orchestra.run.smithi098.stdout:5082 read 0x7cb00f4 thru 0x7cb05b6 (0x4c3 bytes) 2024-09-16T13:18:24.115 INFO:teuthology.orchestra.run.smithi098.stdout:5083 read 0x45fc90f thru 0x4602aed (0x61df bytes) 2024-09-16T13:18:24.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:24.130 INFO:teuthology.orchestra.run.smithi098.stdout:5087 trunc from 0xead8f03 to 0x4f25662 2024-09-16T13:18:24.164 INFO:teuthology.orchestra.run.smithi098.stdout:5089 trunc from 0x4f25662 to 0xcd5589 2024-09-16T13:18:24.194 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:24.184+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:24.216 INFO:teuthology.orchestra.run.smithi098.stdout:5090 read 0x29f162 thru 0x2a36b2 (0x4551 bytes) 2024-09-16T13:18:24.217 INFO:teuthology.orchestra.run.smithi098.stdout:5091 read 0x19f85b thru 0x1a3d76 (0x451c bytes) 2024-09-16T13:18:24.247 INFO:teuthology.orchestra.run.smithi098.stdout:5092 writesame 0xd0e20e1 thru 0xd0e9e12 data_size 0x19(0x7d32 bytes) 2024-09-16T13:18:24.249 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:24.249 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:24.249 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:24.249 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:24.249 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:24.258 INFO:teuthology.orchestra.run.smithi098.stdout:5094 trunc from 0xd0e9e13 to 0x6968ea7 2024-09-16T13:18:24.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:24.265 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:18:24.271 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.271 INFO:teuthology.orchestra.run.smithi098.stdout:5096 punch from 0xd39d28 to 0xd45587, (0xb85f bytes) 2024-09-16T13:18:24.271 INFO:teuthology.orchestra.run.smithi098.stdout:5097 read 0x882582 thru 0x88cfeb (0xaa6a bytes) 2024-09-16T13:18:24.272 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.272 INFO:teuthology.orchestra.run.smithi098.stdout:5100 read 0x3968e7f thru 0x3973d15 (0xae97 bytes) 2024-09-16T13:18:24.275 INFO:teuthology.orchestra.run.smithi098.stdout:5101 trunc from 0x6968ea7 to 0x44cd406 2024-09-16T13:18:24.286 INFO:teuthology.orchestra.run.smithi098.stdout:5102 writesame 0x30654e5 thru 0x306cb5e data_size 0x1e(0x767a bytes) 2024-09-16T13:18:24.292 INFO:teuthology.orchestra.run.smithi098.stdout:5103 read 0x305174c thru 0x3060dd3 (0xf688 bytes) 2024-09-16T13:18:24.295 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:24.287+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:24.308 INFO:teuthology.orchestra.run.smithi098.stdout:5104 write 0x6310a1d thru 0x631a399 (0x997d bytes) 2024-09-16T13:18:24.314 INFO:teuthology.orchestra.run.smithi098.stdout:5106 read 0x51f7c55 thru 0x51fb531 (0x38dd bytes) 2024-09-16T13:18:24.314 INFO:teuthology.orchestra.run.smithi098.stdout:5107 write 0x3d805fc thru 0x3d86201 (0x5c06 bytes) 2024-09-16T13:18:24.324 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.324 INFO:teuthology.orchestra.run.smithi098.stdout:5110 write 0x19e28ba thru 0x19e5b13 (0x325a bytes) 2024-09-16T13:18:24.328 INFO:teuthology.orchestra.run.smithi098.stdout:5111 write 0x56797a6 thru 0x567dbe5 (0x4440 bytes) 2024-09-16T13:18:24.353 INFO:teuthology.orchestra.run.smithi098.stdout:5113 read 0x1125c07 thru 0x1131d7b (0xc175 bytes) 2024-09-16T13:18:24.354 INFO:teuthology.orchestra.run.smithi098.stdout:5114 punch from 0x1603382 to 0x1605cbf, (0x293d bytes) 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.350130863999993, 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.363 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:18:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.349345365000005, 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:18:24.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:18:24.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:24.372 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.348178442999995, 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:18:24.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:18:24.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.342290071999997, 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:18:24.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:18:24.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 79.550936980000003, 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.381 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:18:24.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:18:24.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 72.298834677000002, 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:18:24.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 72.298225236999997, 2024-09-16T13:18:24.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:18:24.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:18:24.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 71.800941422999998, 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:18:24.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:18:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 71.799840606999993, 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:18:24.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:24.403 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 71.799700967000007, 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:24.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:18:24.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 71.799200436999996, 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:24.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 66.804031050999996, 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:18:24.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:18:24.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.547617209999999, 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:18:24.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 55.935054102999999, 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:18:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:18:24.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 29.096947932999999, 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:18:24.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:18:24.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:18:24.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:18:24.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:18:24.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:18:24.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:18:24.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.7328206, 2024-09-16T13:18:24.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:18:24.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.434 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:24.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:24.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:18:24.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:75420 3.4 e88/23 3:2605e3b8:::rbd_object_map.112d9326a5dd.0000000000000026:head v 88'165, mlcod=88'163)", 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.8540915839999998, 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055875079999999997, 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:24.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503433+0000", 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503451+0000", 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7640999999999999e-05 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503476+0000", 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5599999999999999e-05 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503495+0000", 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.825e-05 2024-09-16T13:18:24.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503611+0000", 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011631799999999999 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503644+0000", 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3476000000000003e-05 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.508954+0000", 2024-09-16T13:18:24.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053093799999999998 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.509019+0000", 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5425999999999993e-05 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:80574 3.4 e90/23 3:24bef2c8:::rbd_data.112da28a57db.0000000000000020:head v 90'178, mlcod=89'174)", 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:22.376510+0000", 2024-09-16T13:18:24.444 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 1.981013001, 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034412169999999999, 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376510+0000", 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376510+0000", 2024-09-16T13:18:24.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376512+0000", 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.463e-06 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376524+0000", 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2249000000000001e-05 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376578+0000", 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3672999999999999e-05 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376591+0000", 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3358e-05 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376659+0000", 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7492999999999999e-05 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376690+0000", 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1378999999999999e-05 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.379911+0000", 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003220728 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.379952+0000", 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0874000000000002e-05 2024-09-16T13:18:24.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:82268 3.4 e90/23 3:27686806:::rbd_data.112d12abb20f.000000000000001b:head v 90'182, mlcod=90'180)", 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:23.720888+0000", 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.63663546900000001, 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003148812, 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720888+0000", 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720888+0000", 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720889+0000", 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.344e-06 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720900+0000", 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0942e-05 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720924+0000", 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4128999999999999e-05 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720935+0000", 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1141000000000001e-05 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.721028+0000", 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2785e-05 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.721063+0000", 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4375000000000002e-05 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.723978+0000", 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029157710000000002 2024-09-16T13:18:24.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.724037+0000", 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8325000000000001e-05 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:82956 3.0 e90/23 3:02b9fb4a:::rbd_data.112d12abb20f.0000000000000085:head v 90'317, mlcod=90'315)", 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.22268058499999999, 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035271679999999998, 2024-09-16T13:18:24.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4700000000000001e-07 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134851+0000", 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7770000000000002e-06 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134863+0000", 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1705000000000001e-05 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:24.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134869+0000", 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6829999999999996e-06 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134994+0000", 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012497300000000001 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.135038+0000", 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4548000000000001e-05 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.138295+0000", 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032565070000000001 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.138370+0000", 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5228000000000005e-05 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:24.457 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:24.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:24.459 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:24.451+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:24.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:24.461 INFO:teuthology.orchestra.run.smithi098.stdout:5115 write 0xb2a9e98 thru 0xb2b8061 (0xe1ca bytes) 2024-09-16T13:18:24.461 INFO:teuthology.orchestra.run.smithi098.stdout:5116 read 0x8a8e928 thru 0x8a97342 (0x8a1b bytes) 2024-09-16T13:18:24.461 INFO:teuthology.orchestra.run.smithi098.stdout:5117 write 0x392d064 thru 0x393233d (0x52da bytes) 2024-09-16T13:18:24.461 INFO:teuthology.orchestra.run.smithi098.stdout:5118 writesame 0x1e4abdc thru 0x1e530ea data_size 0x1(0x850f bytes) 2024-09-16T13:18:24.461 INFO:teuthology.orchestra.run.smithi098.stdout:5119 write 0xae1c584 thru 0xae22660 (0x60dd bytes) 2024-09-16T13:18:24.461 INFO:teuthology.orchestra.run.smithi098.stdout:5120 write 0xe5eeea6 thru 0xe5fc724 (0xd87f bytes) 2024-09-16T13:18:24.461 INFO:teuthology.orchestra.run.smithi098.stdout:5121 punch from 0xc792b2a to 0xc79a138, (0x760e bytes) 2024-09-16T13:18:24.461 INFO:teuthology.orchestra.run.smithi098.stdout:5122 trunc from 0xe5fc725 to 0xe0cc1e7 2024-09-16T13:18:24.476 INFO:teuthology.orchestra.run.smithi098.stdout:5126 punch from 0xe0523e4 to 0xe061081, (0xec9d bytes) 2024-09-16T13:18:24.476 INFO:teuthology.orchestra.run.smithi098.stdout:5127 write 0xb3c8d1f thru 0xb3cb0d7 (0x23b9 bytes) 2024-09-16T13:18:24.485 INFO:teuthology.orchestra.run.smithi098.stdout:5128 read 0x972f290 thru 0x97331ec (0x3f5d bytes) 2024-09-16T13:18:24.485 INFO:teuthology.orchestra.run.smithi098.stdout:5129 write 0x73de56a thru 0x73ec5cb (0xe062 bytes) 2024-09-16T13:18:24.495 INFO:teuthology.orchestra.run.smithi098.stdout:5130 read 0x6aa7e77 thru 0x6aa95de (0x1768 bytes) 2024-09-16T13:18:24.495 INFO:teuthology.orchestra.run.smithi098.stdout:5132 read 0x7fd59f thru 0x802351 (0x4db3 bytes) 2024-09-16T13:18:24.497 INFO:teuthology.orchestra.run.smithi098.stdout:5133 read 0x77b4819 thru 0x77b8500 (0x3ce8 bytes) 2024-09-16T13:18:24.497 INFO:teuthology.orchestra.run.smithi098.stdout:5134 writesame 0x757d1f4 thru 0x757f241 data_size 0x5(0x204e bytes) 2024-09-16T13:18:24.501 INFO:teuthology.orchestra.run.smithi098.stdout:5135 write 0x9c47835 thru 0x9c4a913 (0x30df bytes) 2024-09-16T13:18:24.507 INFO:teuthology.orchestra.run.smithi098.stdout:5136 read 0x76caf70 thru 0x76d10a0 (0x6131 bytes) 2024-09-16T13:18:24.507 INFO:teuthology.orchestra.run.smithi098.stdout:5138 write 0x36597c3 thru 0x36672d2 (0xdb10 bytes) 2024-09-16T13:18:24.512 INFO:teuthology.orchestra.run.smithi098.stdout:5139 read 0x653f47f thru 0x65441b3 (0x4d35 bytes) 2024-09-16T13:18:24.512 INFO:teuthology.orchestra.run.smithi098.stdout:5140 punch from 0xb6ba835 to 0xb6c9d14, (0xf4df bytes) 2024-09-16T13:18:24.512 INFO:teuthology.orchestra.run.smithi098.stdout:5141 writesame 0xae7155f thru 0xae7a42b data_size 0x1(0x8ecd bytes) 2024-09-16T13:18:24.543 INFO:teuthology.orchestra.run.smithi098.stdout:5142 read 0x6ec51a5 thru 0x6ec9edc (0x4d38 bytes) 2024-09-16T13:18:24.544 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.544 INFO:teuthology.orchestra.run.smithi098.stdout:5145 read 0x567ff21 thru 0x56801c3 (0x2a3 bytes) 2024-09-16T13:18:24.545 INFO:teuthology.orchestra.run.smithi098.stdout:5146 read 0xc1ecae0 thru 0xc1ed74e (0xc6f bytes) 2024-09-16T13:18:24.545 INFO:teuthology.orchestra.run.smithi098.stdout:5148 read 0x28b2d70 thru 0x28c2cd0 (0xff61 bytes) 2024-09-16T13:18:24.545 INFO:teuthology.orchestra.run.smithi098.stdout:5151 read 0xc5b6624 thru 0xc5c09bd (0xa39a bytes) 2024-09-16T13:18:24.545 INFO:teuthology.orchestra.run.smithi098.stdout:5152 write 0x6e8c201 thru 0x6e92384 (0x6184 bytes) 2024-09-16T13:18:24.553 INFO:teuthology.orchestra.run.smithi098.stdout:5153 write 0x5aad4f9 thru 0x5ab63fb (0x8f03 bytes) 2024-09-16T13:18:24.559 INFO:teuthology.orchestra.run.smithi098.stdout:5154 punch from 0x65a35a4 to 0x65a9e56, (0x68b2 bytes) 2024-09-16T13:18:24.559 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:24.551+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:24.562 INFO:teuthology.orchestra.run.smithi098.stdout:5155 trunc from 0xe0cc1e7 to 0xbca3aaf 2024-09-16T13:18:24.579 INFO:teuthology.orchestra.run.smithi098.stdout:5156 read 0x22aebdd thru 0x22b5e99 (0x72bd bytes) 2024-09-16T13:18:24.580 INFO:teuthology.orchestra.run.smithi098.stdout:5157 write 0xb66f788 thru 0xb67a22d (0xaaa6 bytes) 2024-09-16T13:18:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:24.588 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:24.588 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:5160 punch from 0x1c55b11 to 0x1c5c752, (0x6c41 bytes) 2024-09-16T13:18:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:5162 punch from 0x8081280 to 0x808a252, (0x8fd2 bytes) 2024-09-16T13:18:24.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:24.603 INFO:teuthology.orchestra.run.smithi098.stdout:5164 trunc from 0xbca3aaf to 0x1e17987 2024-09-16T13:18:24.625 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:24.647 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:24.653 INFO:teuthology.orchestra.run.smithi098.stdout:5165 writesame 0xbebc9e1 thru 0xbec6026 data_size 0x5(0x9646 bytes) 2024-09-16T13:18:24.653 INFO:teuthology.orchestra.run.smithi098.stdout:5166 write 0x7d0a79d thru 0x7d110f9 (0x695d bytes) 2024-09-16T13:18:24.660 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:24.652+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:24.669 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:24.673 INFO:teuthology.orchestra.run.smithi098.stdout:5167 write 0xd6c7d20 thru 0xd6c8d72 (0x1053 bytes) 2024-09-16T13:18:24.679 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.697 INFO:teuthology.orchestra.run.smithi098.stdout:5169 trunc from 0xd6c8d73 to 0x2358afb 2024-09-16T13:18:24.701 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:24.704 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:24.704 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:24.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:24.718 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.727 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:24.739 INFO:teuthology.orchestra.run.smithi098.stdout:5171 write 0x7ee24d3 thru 0x7ee6ca4 (0x47d2 bytes) 2024-09-16T13:18:24.744 INFO:teuthology.orchestra.run.smithi098.stdout:5172 read 0x68ddc8 thru 0x694a36 (0x6c6f bytes) 2024-09-16T13:18:24.746 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.747 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:24.761 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:24.752+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:24.762 INFO:teuthology.orchestra.run.smithi098.stdout:5174 writesame 0xa166bb9 thru 0xa170491 data_size 0x1(0x98d9 bytes) 2024-09-16T13:18:24.762 INFO:teuthology.orchestra.run.smithi098.stdout:5175 punch from 0x205aec1 to 0x2069193, (0xe2d2 bytes) 2024-09-16T13:18:24.769 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:24.770 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:24.770 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:24.770 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:24.770 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:24.771 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:24.771 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:24.771 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:24.771 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:24.773 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:24.774 INFO:teuthology.orchestra.run.smithi098.stdout:5176 writesame 0xacd4132 thru 0xacd4539 data_size 0xc(0x408 bytes) 2024-09-16T13:18:24.774 INFO:teuthology.orchestra.run.smithi098.stdout:5177 writesame 0x1f4124c thru 0x1f42582 data_size 0x1(0x1337 bytes) 2024-09-16T13:18:24.782 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.782 INFO:teuthology.orchestra.run.smithi098.stdout:5180 punch from 0xf64cec to 0xf6550e, (0x822 bytes) 2024-09-16T13:18:24.782 INFO:teuthology.orchestra.run.smithi098.stdout:5181 punch from 0xa14ee57 to 0xa158e5e, (0xa007 bytes) 2024-09-16T13:18:24.782 INFO:teuthology.orchestra.run.smithi098.stdout:5182 writesame 0x12f599a thru 0x12f9d37 data_size 0xf(0x439e bytes) 2024-09-16T13:18:24.787 INFO:teuthology.orchestra.run.smithi098.stdout:5183 writesame 0x82f2173 thru 0x8301d2c data_size 0x7(0xfbba bytes) 2024-09-16T13:18:24.800 INFO:teuthology.orchestra.run.smithi098.stdout:5184 punch from 0x24e73da to 0x24f2751, (0xb377 bytes) 2024-09-16T13:18:24.800 INFO:teuthology.orchestra.run.smithi098.stdout:5187 write 0x8cb8292 thru 0x8cbeff8 (0x6d67 bytes) 2024-09-16T13:18:24.822 INFO:teuthology.orchestra.run.smithi098.stdout:5188 write 0xdbbb2f2 thru 0xdbc219b (0x6eaa bytes) 2024-09-16T13:18:24.826 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:24.826 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:24.831 INFO:teuthology.orchestra.run.smithi098.stdout:5190 punch from 0xcb01b59 to 0xcb07e69, (0x6310 bytes) 2024-09-16T13:18:24.831 INFO:teuthology.orchestra.run.smithi098.stdout:5191 write 0x8b23a5a thru 0x8b2b00c (0x75b3 bytes) 2024-09-16T13:18:24.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:24.838 INFO:teuthology.orchestra.run.smithi098.stdout:5194 read 0xc6e51db thru 0xc6eacc7 (0x5aed bytes) 2024-09-16T13:18:24.839 INFO:teuthology.orchestra.run.smithi098.stdout:5195 read 0x7c9c365 thru 0x7ca6092 (0x9d2e bytes) 2024-09-16T13:18:24.839 INFO:teuthology.orchestra.run.smithi098.stdout:5198 read 0x46f773d thru 0x4704874 (0xd138 bytes) 2024-09-16T13:18:24.839 INFO:teuthology.orchestra.run.smithi098.stdout:5200 read 0x41f5b1 thru 0x4219da (0x242a bytes) 2024-09-16T13:18:24.840 INFO:teuthology.orchestra.run.smithi098.stdout:5201 writesame 0x534c845 thru 0x53539be data_size 0xe(0x717a bytes) 2024-09-16T13:18:24.849 INFO:teuthology.orchestra.run.smithi098.stdout:5202 writesame 0x7434c9c thru 0x743871a data_size 0x5(0x3a7f bytes) 2024-09-16T13:18:24.858 INFO:teuthology.orchestra.run.smithi098.stdout:5203 write 0x1ebe5f1 thru 0x1ecc1d0 (0xdbe0 bytes) 2024-09-16T13:18:24.861 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:24.852+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:24.864 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:24.864 INFO:teuthology.orchestra.run.smithi098.stdout:5205 write 0x3c21055 thru 0x3c26d3f (0x5ceb bytes) 2024-09-16T13:18:24.881 INFO:teuthology.orchestra.run.smithi098.stdout:5206 trunc from 0xdbc219c to 0x767c19e 2024-09-16T13:18:24.905 INFO:teuthology.orchestra.run.smithi098.stdout:5207 write 0x3df89a5 thru 0x3dfd8e9 (0x4f45 bytes) 2024-09-16T13:18:24.920 INFO:teuthology.orchestra.run.smithi098.stdout:5208 trunc from 0x767c19e to 0xbb6f8d0 2024-09-16T13:18:24.932 INFO:teuthology.orchestra.run.smithi098.stdout:5210 punch from 0x37a0564 to 0x37a9f39, (0x99d5 bytes) 2024-09-16T13:18:24.932 INFO:teuthology.orchestra.run.smithi098.stdout:5211 clone 37 order 20 su 65536 sc 15 2024-09-16T13:18:24.952 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:24.952 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:24.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:24.962 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:24.955+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:25.051 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:25.051 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:25.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:25.063 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:25.056+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:25.156 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:25.156 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:25.164 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:25.155+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:25.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:25.227 INFO:teuthology.orchestra.run.smithi151.stderr:Nothing to do for non-filestore backend 2024-09-16T13:18:25.264 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:25.259+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:25.266 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:25.266 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:25.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:25.366 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:25.359+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:25.425 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:25.425 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:25.426 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone36 from 0xbb6f8d0 (overlap 0xcd5589) to 0x0 2024-09-16T13:18:25.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:25.481 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:25.475+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:25.533 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:25.534 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:25.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:25.568 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:25.559+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:25.628 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #35, image image_client.0-clone35 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent36 2024-09-16T13:18:25.646 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:25.646 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:25.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:25.693 INFO:teuthology.orchestra.run.smithi098.stdout:5213 write 0xe538958 thru 0xe5420d9 (0x9782 bytes) 2024-09-16T13:18:25.701 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:25.695+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:25.702 INFO:teuthology.orchestra.run.smithi098.stdout:5214 write 0x7f81d31 thru 0x7f8671b (0x49eb bytes) 2024-09-16T13:18:25.708 INFO:teuthology.orchestra.run.smithi098.stdout:5215 writesame 0xcc46399 thru 0xcc487b6 data_size 0x17(0x241e bytes) 2024-09-16T13:18:25.715 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:25.715 INFO:teuthology.orchestra.run.smithi098.stdout:5220 read 0x21ff1a0 thru 0x220377d (0x45de bytes) 2024-09-16T13:18:25.715 INFO:teuthology.orchestra.run.smithi098.stdout:5221 write 0x182bf6b thru 0x182ea93 (0x2b29 bytes) 2024-09-16T13:18:25.723 INFO:teuthology.orchestra.run.smithi098.stdout:5222 read 0x8e91f4f thru 0x8e9caf0 (0xaba2 bytes) 2024-09-16T13:18:25.723 INFO:teuthology.orchestra.run.smithi098.stdout:5223 read 0x7dc0eaa thru 0x7dc6cc4 (0x5e1b bytes) 2024-09-16T13:18:25.723 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:25.724 INFO:teuthology.orchestra.run.smithi098.stdout:5225 punch from 0x9712acf to 0x971713e, (0x466f bytes) 2024-09-16T13:18:25.724 INFO:teuthology.orchestra.run.smithi098.stdout:5226 write 0x75d4e82 thru 0x75db106 (0x6285 bytes) 2024-09-16T13:18:25.757 INFO:teuthology.orchestra.run.smithi098.stdout:5227 trunc from 0xe5420da to 0x32c4aba 2024-09-16T13:18:25.762 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:25.763 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:25.769 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:25.763+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:25.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:25.807 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:25.807 INFO:teuthology.orchestra.run.smithi098.stdout:5229 punch from 0x2339e7f to 0x2341b5d, (0x7cde bytes) 2024-09-16T13:18:25.819 INFO:teuthology.orchestra.run.smithi098.stdout:5230 write 0x41d1917 thru 0x41d983f (0x7f29 bytes) 2024-09-16T13:18:25.826 INFO:teuthology.orchestra.run.smithi098.stdout:5234 read 0x38ba1a thru 0x3961f7 (0xa7de bytes) 2024-09-16T13:18:25.827 INFO:teuthology.orchestra.run.smithi098.stdout:5235 read 0x1e276ce thru 0x1e29356 (0x1c89 bytes) 2024-09-16T13:18:25.828 INFO:teuthology.orchestra.run.smithi098.stdout:5236 trunc from 0x41d9840 to 0x3dc6582 2024-09-16T13:18:25.861 INFO:teuthology.orchestra.run.smithi098.stdout:5237 trunc from 0x3dc6582 to 0xb57f1dd 2024-09-16T13:18:25.870 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:25.860+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:25.883 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:25.883 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:25.890 INFO:teuthology.orchestra.run.smithi098.stdout:5238 write 0xdfc21c4 thru 0xdfc935b (0x7198 bytes) 2024-09-16T13:18:25.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:25.896 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:25.896 INFO:teuthology.orchestra.run.smithi098.stdout:5240 punch from 0xe16b48 to 0xe1e700, (0x7bb8 bytes) 2024-09-16T13:18:25.896 INFO:teuthology.orchestra.run.smithi098.stdout:5241 write 0x6b8c72a thru 0x6b8e140 (0x1a17 bytes) 2024-09-16T13:18:25.926 INFO:teuthology.orchestra.run.smithi098.stdout:5242 trunc from 0xdfc935c to 0x2cb759f 2024-09-16T13:18:25.959 INFO:teuthology.orchestra.run.smithi098.stdout:5243 punch from 0x28edd1c to 0x28f3695, (0x5979 bytes) 2024-09-16T13:18:25.964 INFO:teuthology.orchestra.run.smithi098.stdout:5244 trunc from 0x2cb759f to 0x60a203c 2024-09-16T13:18:25.971 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:25.964+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:25.977 INFO:teuthology.orchestra.run.smithi098.stdout:5246 punch from 0x49edb89 to 0x49fac05, (0xd07c bytes) 2024-09-16T13:18:25.977 INFO:teuthology.orchestra.run.smithi098.stdout:5247 writesame 0x1e4f8f1 thru 0x1e5603c data_size 0x16(0x674c bytes) 2024-09-16T13:18:25.985 INFO:teuthology.orchestra.run.smithi098.stdout:5251 read 0x432bbca thru 0x4337636 (0xba6d bytes) 2024-09-16T13:18:25.986 INFO:teuthology.orchestra.run.smithi098.stdout:5252 read 0x4f22454 thru 0x4f2c0ef (0x9c9c bytes) 2024-09-16T13:18:25.986 INFO:teuthology.orchestra.run.smithi098.stdout:5253 write 0x1549959 thru 0x154eecb (0x5573 bytes) 2024-09-16T13:18:25.993 INFO:teuthology.orchestra.run.smithi098.stdout:5257 punch from 0x926d20 to 0x92d2b5, (0x6595 bytes) 2024-09-16T13:18:25.993 INFO:teuthology.orchestra.run.smithi098.stdout:5258 read 0xcbc3ab thru 0xcc931c (0xcf72 bytes) 2024-09-16T13:18:25.994 INFO:teuthology.orchestra.run.smithi098.stdout:5260 write 0x2984038 thru 0x2989f26 (0x5eef bytes) 2024-09-16T13:18:26.011 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:26.011 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:26.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:26.021 INFO:teuthology.orchestra.run.smithi098.stdout:5261 writesame 0xc7a3a5f thru 0xc7a4b8f data_size 0x9(0x1131 bytes) 2024-09-16T13:18:26.021 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:26.022 INFO:teuthology.orchestra.run.smithi098.stdout:5263 read 0xbccdfee thru 0xbcd5df7 (0x7e0a bytes) 2024-09-16T13:18:26.023 INFO:teuthology.orchestra.run.smithi098.stdout:5264 trunc from 0xc7a4b90 to 0xaf78888 2024-09-16T13:18:26.036 INFO:teuthology.orchestra.run.smithi098.stdout:5265 write 0x6807bf0 thru 0x6808b2f (0xf40 bytes) 2024-09-16T13:18:26.043 INFO:teuthology.orchestra.run.smithi098.stdout:5266 read 0x2a79cc5 thru 0x2a80e06 (0x7142 bytes) 2024-09-16T13:18:26.060 INFO:teuthology.orchestra.run.smithi098.stdout:5268 writesame 0xc437436 thru 0xc43c84d data_size 0x68(0x5418 bytes) 2024-09-16T13:18:26.060 INFO:teuthology.orchestra.run.smithi098.stdout:5269 write 0x4ee3d54 thru 0x4ee6a5d (0x2d0a bytes) 2024-09-16T13:18:26.066 INFO:teuthology.orchestra.run.smithi098.stdout:5271 write 0xdd1180 thru 0xdda997 (0x9818 bytes) 2024-09-16T13:18:26.071 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:26.064+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:26.076 INFO:teuthology.orchestra.run.smithi098.stdout:5272 write 0x3c75046 thru 0x3c7762c (0x25e7 bytes) 2024-09-16T13:18:26.093 INFO:teuthology.orchestra.run.smithi098.stdout:5273 trunc from 0xc43c84e to 0x59593c3 2024-09-16T13:18:26.115 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:26.115 INFO:teuthology.orchestra.run.smithi098.stdout:5275 read 0x49290a7 thru 0x492cbbb (0x3b15 bytes) 2024-09-16T13:18:26.126 INFO:teuthology.orchestra.run.smithi098.stdout:5277 write 0x61ce129 thru 0x61d097c (0x2854 bytes) 2024-09-16T13:18:26.130 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:26.130 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:26.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:26.162 INFO:teuthology.orchestra.run.smithi098.stdout:5278 write 0xe17bf3c thru 0xe17d385 (0x144a bytes) 2024-09-16T13:18:26.175 INFO:teuthology.orchestra.run.smithi098.stdout:5281 trunc from 0xe17d386 to 0xb4b6f72 2024-09-16T13:18:26.177 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:26.168+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:26.197 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:26.197 INFO:teuthology.orchestra.run.smithi098.stdout:5284 punch from 0x3bec86c to 0x3bf2ffa, (0x678e bytes) 2024-09-16T13:18:26.197 INFO:teuthology.orchestra.run.smithi098.stdout:5286 punch from 0x78d338f to 0x78d39b6, (0x627 bytes) 2024-09-16T13:18:26.215 INFO:teuthology.orchestra.run.smithi098.stdout:5288 write 0xe113520 thru 0xe116c46 (0x3727 bytes) 2024-09-16T13:18:26.223 INFO:teuthology.orchestra.run.smithi098.stdout:5289 writesame 0xb7b172b thru 0xb7b732e data_size 0x4e(0x5c04 bytes) 2024-09-16T13:18:26.229 INFO:teuthology.orchestra.run.smithi098.stdout:5290 write 0x73f235f thru 0x73f8e5e (0x6b00 bytes) 2024-09-16T13:18:26.247 INFO:teuthology.orchestra.run.smithi098.stdout:5291 write 0xe32e483 thru 0xe32e845 (0x3c3 bytes) 2024-09-16T13:18:26.253 INFO:teuthology.orchestra.run.smithi098.stdout:5292 read 0xb9de2d3 thru 0xb9ed95c (0xf68a bytes) 2024-09-16T13:18:26.253 INFO:teuthology.orchestra.run.smithi098.stdout:5293 writesame 0x5011eaa thru 0x50210fa data_size 0x29(0xf251 bytes) 2024-09-16T13:18:26.272 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:26.272 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:26.273 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:26.264+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:26.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:18:26.283 INFO:teuthology.orchestra.run.smithi098.stdout:5295 trunc from 0xe32e846 to 0x1acf0b6 2024-09-16T13:18:26.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:26.335 INFO:teuthology.orchestra.run.smithi098.stdout:5297 read 0x509583 thru 0x510433 (0x6eb1 bytes) 2024-09-16T13:18:26.336 INFO:teuthology.orchestra.run.smithi098.stdout:5298 write 0xafbb01 thru 0xb06580 (0xaa80 bytes) 2024-09-16T13:18:26.345 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:26.362 INFO:teuthology.orchestra.run.smithi098.stdout:5300 write 0x4ed4f50 thru 0x4ed8ff6 (0x40a7 bytes) 2024-09-16T13:18:26.375 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:26.368+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:26.391 INFO:teuthology.orchestra.run.smithi098.stdout:5301 writesame 0xc700768 thru 0xc70b16c data_size 0x5(0xaa05 bytes) 2024-09-16T13:18:26.396 INFO:teuthology.orchestra.run.smithi098.stdout:5302 trunc from 0xc70b16d to 0x8d29508 2024-09-16T13:18:26.406 INFO:teuthology.orchestra.run.smithi098.stdout:5303 punch from 0x24efbe to 0x25aab5, (0xbaf7 bytes) 2024-09-16T13:18:26.406 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:26.407 INFO:teuthology.orchestra.run.smithi098.stdout:5306 read 0x551b4d6 thru 0x552220e (0x6d39 bytes) 2024-09-16T13:18:26.407 INFO:teuthology.orchestra.run.smithi098.stdout:5307 read 0x892d977 thru 0x893247e (0x4b08 bytes) 2024-09-16T13:18:26.426 INFO:teuthology.orchestra.run.smithi098.stdout:5308 write 0xdaf14a4 thru 0xdb0052d (0xf08a bytes) 2024-09-16T13:18:26.435 INFO:teuthology.orchestra.run.smithi098.stdout:5309 read 0x437214 thru 0x446584 (0xf371 bytes) 2024-09-16T13:18:26.439 INFO:teuthology.orchestra.run.smithi098.stdout:5312 trunc from 0xdb0052e to 0xc6966f2 2024-09-16T13:18:26.460 INFO:teuthology.orchestra.run.smithi098.stdout:5313 write 0xc1ef242 thru 0xc1fca74 (0xd833 bytes) 2024-09-16T13:18:26.460 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:26.460 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:26.469 INFO:teuthology.orchestra.run.smithi098.stdout:5314 punch from 0x22160e6 to 0x221dc52, (0x7b6c bytes) 2024-09-16T13:18:26.469 INFO:teuthology.orchestra.run.smithi098.stdout:5316 read 0xa24076e thru 0xa24359b (0x2e2e bytes) 2024-09-16T13:18:26.469 INFO:teuthology.orchestra.run.smithi098.stdout:5317 read 0x3f002a7 thru 0x3f048c2 (0x461c bytes) 2024-09-16T13:18:26.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:26.475 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:26.468+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:26.480 INFO:teuthology.orchestra.run.smithi098.stdout:5318 writesame 0xcb76eed thru 0xcb806e8 data_size 0x47(0x97fc bytes) 2024-09-16T13:18:26.481 INFO:teuthology.orchestra.run.smithi098.stdout:5319 writesame 0x3a4917c thru 0x3a51912 data_size 0x1(0x8797 bytes) 2024-09-16T13:18:26.519 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:26.534 INFO:teuthology.orchestra.run.smithi098.stdout:5321 trunc from 0xcb806e9 to 0x21f165e 2024-09-16T13:18:26.558 INFO:teuthology.orchestra.run.smithi098.stdout:5323 read 0x1dc291e thru 0x1dd03e6 (0xdac9 bytes) 2024-09-16T13:18:26.575 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:26.575 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:26.576 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:26.568+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:26.576 INFO:teuthology.orchestra.run.smithi098.stdout:5324 writesame 0x52342ed thru 0x5240f6b data_size 0x1(0xcc7f bytes) 2024-09-16T13:18:26.576 INFO:teuthology.orchestra.run.smithi098.stdout:5326 write 0x4425116 thru 0x44318a4 (0xc78f bytes) 2024-09-16T13:18:26.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:26.588 INFO:teuthology.orchestra.run.smithi098.stdout:5327 trunc from 0x5240f6c to 0x35de598 2024-09-16T13:18:26.607 INFO:teuthology.orchestra.run.smithi098.stdout:5328 read 0x1bc089f thru 0x1bc37fe (0x2f60 bytes) 2024-09-16T13:18:26.608 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:26.635 INFO:teuthology.orchestra.run.smithi098.stdout:5331 write 0xb87f0c0 thru 0xb87f6d1 (0x612 bytes) 2024-09-16T13:18:26.641 INFO:teuthology.orchestra.run.smithi098.stdout:5332 write 0x3149d5c thru 0x314d4d9 (0x377e bytes) 2024-09-16T13:18:26.654 INFO:teuthology.orchestra.run.smithi098.stdout:5333 trunc from 0xb87f6d2 to 0x7679a28 2024-09-16T13:18:26.675 INFO:teuthology.orchestra.run.smithi098.stdout:5334 write 0x7c941d thru 0x7cd0a4 (0x3c88 bytes) 2024-09-16T13:18:26.677 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:26.668+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:26.684 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:26.684 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:26.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:26.711 INFO:teuthology.orchestra.run.smithi098.stdout:5335 writesame 0xc80c241 thru 0xc814943 data_size 0x29(0x8703 bytes) 2024-09-16T13:18:26.711 INFO:teuthology.orchestra.run.smithi098.stdout:5336 read 0x27ad39c thru 0x27b542f (0x8094 bytes) 2024-09-16T13:18:26.711 INFO:teuthology.orchestra.run.smithi098.stdout:5339 read 0x2fd3c1b thru 0x2fd91db (0x55c1 bytes) 2024-09-16T13:18:26.711 INFO:teuthology.orchestra.run.smithi098.stdout:5340 read 0xa7671e3 thru 0xa7745ce (0xd3ec bytes) 2024-09-16T13:18:26.711 INFO:teuthology.orchestra.run.smithi098.stdout:5341 write 0x355edc8 thru 0x356379a (0x49d3 bytes) 2024-09-16T13:18:26.719 INFO:teuthology.orchestra.run.smithi098.stdout:5343 read 0x8c172e4 thru 0x8c1ac0e (0x392b bytes) 2024-09-16T13:18:26.719 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:26.720 INFO:teuthology.orchestra.run.smithi098.stdout:5346 read 0x90cbeff thru 0x90db070 (0xf172 bytes) 2024-09-16T13:18:26.720 INFO:teuthology.orchestra.run.smithi098.stdout:5347 write 0x84c6036 thru 0x84d2fad (0xcf78 bytes) 2024-09-16T13:18:26.736 INFO:teuthology.orchestra.run.smithi098.stdout:5348 trunc from 0xc814944 to 0x809c722 2024-09-16T13:18:26.751 INFO:teuthology.orchestra.run.smithi098.stdout:5349 write 0x25afe0b thru 0x25b81f2 (0x83e8 bytes) 2024-09-16T13:18:26.759 INFO:teuthology.orchestra.run.smithi098.stdout:5351 punch from 0x46bf0a1 to 0x46c6d01, (0x7c60 bytes) 2024-09-16T13:18:26.771 INFO:teuthology.orchestra.run.smithi098.stdout:5352 write 0x8c2346f thru 0x8c2e271 (0xae03 bytes) 2024-09-16T13:18:26.775 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:26.778 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:26.768+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:26.780 INFO:teuthology.orchestra.run.smithi098.stdout:5353 write 0x3911895 thru 0x39146bc (0x2e28 bytes) 2024-09-16T13:18:26.790 INFO:teuthology.orchestra.run.smithi098.stdout:5354 trunc from 0x8c2e272 to 0x575c2fb 2024-09-16T13:18:26.810 INFO:teuthology.orchestra.run.smithi098.stdout:5355 write 0x48c958 thru 0x491c5a (0x5303 bytes) 2024-09-16T13:18:26.817 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:26.817 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:26.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:26.847 INFO:teuthology.orchestra.run.smithi098.stdout:5356 write 0xea8e2c6 thru 0xea99187 (0xaec2 bytes) 2024-09-16T13:18:26.855 INFO:teuthology.orchestra.run.smithi098.stdout:5357 write 0x7e23f45 thru 0x7e31e73 (0xdf2f bytes) 2024-09-16T13:18:26.862 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:26.862 INFO:teuthology.orchestra.run.smithi098.stdout:5359 read 0x266ec1 thru 0x26fa8a (0x8bca bytes) 2024-09-16T13:18:26.863 INFO:teuthology.orchestra.run.smithi098.stdout:5360 read 0x8264126 thru 0x8265acd (0x19a8 bytes) 2024-09-16T13:18:26.863 INFO:teuthology.orchestra.run.smithi098.stdout:5361 punch from 0x35cefb2 to 0x35dbdf7, (0xce45 bytes) 2024-09-16T13:18:26.863 INFO:teuthology.orchestra.run.smithi098.stdout:5363 write 0x7d31a95 thru 0x7d37d97 (0x6303 bytes) 2024-09-16T13:18:26.865 INFO:teuthology.orchestra.run.smithi098.stdout:5364 read 0xa624f1f thru 0xa62f599 (0xa67b bytes) 2024-09-16T13:18:26.865 INFO:teuthology.orchestra.run.smithi098.stdout:5366 writesame 0x2b3528b thru 0x2b3f4a6 data_size 0xa(0xa21c bytes) 2024-09-16T13:18:26.872 INFO:teuthology.orchestra.run.smithi098.stdout:5367 read 0x39e7bd4 thru 0x39ed3ad (0x57da bytes) 2024-09-16T13:18:26.878 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:26.868+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:26.890 INFO:teuthology.orchestra.run.smithi098.stdout:5368 trunc from 0xea99188 to 0x3113a67 2024-09-16T13:18:26.914 INFO:teuthology.orchestra.run.smithi098.stdout:5369 read 0x101ff67 thru 0x10219b7 (0x1a51 bytes) 2024-09-16T13:18:26.937 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:26.945 INFO:teuthology.orchestra.run.smithi098.stdout:5370 write 0xeaf86cb thru 0xeb00422 (0x7d58 bytes) 2024-09-16T13:18:26.953 INFO:teuthology.orchestra.run.smithi098.stdout:5371 write 0x1d1d092 thru 0x1d26ab3 (0x9a22 bytes) 2024-09-16T13:18:26.956 INFO:teuthology.orchestra.run.smithi098.stdout:5372 read 0x6a93a46 thru 0x6a9f8da (0xbe95 bytes) 2024-09-16T13:18:26.957 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:26.957 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:26.957 INFO:teuthology.orchestra.run.smithi098.stdout:5374 read 0x9143e97 thru 0x914f93e (0xbaa8 bytes) 2024-09-16T13:18:26.961 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:26.969 INFO:teuthology.orchestra.run.smithi098.stdout:5375 trunc from 0xeb00423 to 0x7b79767 2024-09-16T13:18:26.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:26.981 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:26.987 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:26.980+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:26.989 INFO:teuthology.orchestra.run.smithi098.stdout:5377 writesame 0x3bfb8db thru 0x3c03bcc data_size 0x3(0x82f2 bytes) 2024-09-16T13:18:27.003 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:27.007 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.024 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:27.030 INFO:teuthology.orchestra.run.smithi098.stdout:5379 writesame 0xda00185 thru 0xda0ba09 data_size 0x1(0xb885 bytes) 2024-09-16T13:18:27.030 INFO:teuthology.orchestra.run.smithi098.stdout:5381 read 0x3b69183 thru 0x3b6ee4a (0x5cc8 bytes) 2024-09-16T13:18:27.036 INFO:teuthology.orchestra.run.smithi098.stdout:5382 trunc from 0xda0ba0a to 0xa052d5e 2024-09-16T13:18:27.050 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:27.051 INFO:teuthology.orchestra.run.smithi098.stdout:5384 write 0x58a3f51 thru 0x58a9d40 (0x5df0 bytes) 2024-09-16T13:18:27.057 INFO:teuthology.orchestra.run.smithi098.stdout:5385 punch from 0x38959b8 to 0x389fa0d, (0xa055 bytes) 2024-09-16T13:18:27.070 INFO:teuthology.orchestra.run.smithi098.stdout:5386 trunc from 0xa052d5e to 0x2485adf 2024-09-16T13:18:27.074 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:27.074 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:27.074 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:27.080 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:27.081 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:27.075+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:27.093 INFO:teuthology.orchestra.run.smithi098.stdout:5388 write 0x4f6167 thru 0x501362 (0xb1fc bytes) 2024-09-16T13:18:27.102 INFO:teuthology.orchestra.run.smithi098.stdout:5389 read 0x210a192 thru 0x21153af (0xb21e bytes) 2024-09-16T13:18:27.102 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.103 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 83.096185339000002, 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.109 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:18:27.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:18:27.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:18:27.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:18:27.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:18:27.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:18:27.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:18:27.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:18:27.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:18:27.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:18:27.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:18:27.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:18:27.115 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.115 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.115 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.115 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:18:27.115 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:27.115 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 83.095399839999999, 2024-09-16T13:18:27.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:27.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:18:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:18:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:18:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 83.094232918000003, 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:18:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:18:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 83.088344547000005, 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:18:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:18:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:18:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:18:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:18:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:18:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:18:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:18:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 82.296991454999997, 2024-09-16T13:18:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:18:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:18:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:18:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:18:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 75.044889151999996, 2024-09-16T13:18:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:18:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 75.044279712000005, 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:18:27.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:18:27.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:18:27.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:18:27.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:18:27.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 74.546995898000006, 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:18:27.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:18:27.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:18:27.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 74.545895082000001, 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:18:27.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 74.545755442000001, 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:18:27.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.158 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 74.545254912000004, 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:18:27.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:18:27.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:18:27.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:18:27.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:18:27.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.165 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.165 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:18:27.165 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:27.165 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 69.550085526000004, 2024-09-16T13:18:27.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:18:27.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:27.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.167 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:18:27.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:18:27.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:18:27.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:27.170 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 59.293671685, 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:18:27.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:18:27.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:18:27.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:18:27.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.175 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 58.681108578, 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:18:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:18:27.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:18:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:18:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:18:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:18:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:18:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:18:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 31.843002408, 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:27.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:18:27.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:18:27.186 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 31.478875075000001, 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:18:27.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:75420 3.4 e88/23 3:2605e3b8:::rbd_object_map.112d9326a5dd.0000000000000026:head v 88'165, mlcod=88'163)", 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:27.191 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 7.6001460590000001, 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055875079999999997, 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:27.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503433+0000", 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503451+0000", 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7640999999999999e-05 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503476+0000", 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5599999999999999e-05 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503495+0000", 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.825e-05 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503611+0000", 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011631799999999999 2024-09-16T13:18:27.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503644+0000", 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3476000000000003e-05 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.508954+0000", 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053093799999999998 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.509019+0000", 2024-09-16T13:18:27.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5425999999999993e-05 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:80574 3.4 e90/23 3:24bef2c8:::rbd_data.112da28a57db.0000000000000020:head v 90'178, mlcod=89'174)", 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:22.376510+0000", 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.7270674760000002, 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034412169999999999, 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376510+0000", 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376510+0000", 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376512+0000", 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.463e-06 2024-09-16T13:18:27.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376524+0000", 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2249000000000001e-05 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376578+0000", 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3672999999999999e-05 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376591+0000", 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3358e-05 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376659+0000", 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7492999999999999e-05 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376690+0000", 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1378999999999999e-05 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.379911+0000", 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003220728 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.379952+0000", 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0874000000000002e-05 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.200 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:82268 3.4 e90/23 3:27686806:::rbd_data.112d12abb20f.000000000000001b:head v 90'182, mlcod=90'180)", 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:23.720888+0000", 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.382689944, 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003148812, 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720888+0000", 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720888+0000", 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720889+0000", 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.344e-06 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720900+0000", 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0942e-05 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720924+0000", 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4128999999999999e-05 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720935+0000", 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1141000000000001e-05 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.721028+0000", 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2785e-05 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.721063+0000", 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4375000000000002e-05 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.723978+0000", 2024-09-16T13:18:27.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029157710000000002 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.724037+0000", 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8325000000000001e-05 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:82956 3.0 e90/23 3:02b9fb4a:::rbd_data.112d12abb20f.0000000000000085:head v 90'317, mlcod=90'315)", 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 2.9687350600000002, 2024-09-16T13:18:27.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035271679999999998, 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4700000000000001e-07 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134851+0000", 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7770000000000002e-06 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134863+0000", 2024-09-16T13:18:27.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1705000000000001e-05 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134869+0000", 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6829999999999996e-06 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134994+0000", 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012497300000000001 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.135038+0000", 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4548000000000001e-05 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.138295+0000", 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032565070000000001 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.138370+0000", 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5228000000000005e-05 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.209 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.210 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.210 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.210 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.210 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:27.211 INFO:teuthology.orchestra.run.smithi098.stdout:5392 write 0x42a7ef8 thru 0x42ae9a2 (0x6aab bytes) 2024-09-16T13:18:27.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:27.213 INFO:teuthology.orchestra.run.smithi098.stdout:5393 read 0x2b456e9 thru 0x2b4d994 (0x82ac bytes) 2024-09-16T13:18:27.213 INFO:teuthology.orchestra.run.smithi098.stdout:5394 writesame 0x8276310 thru 0x827c134 data_size 0xb(0x5e25 bytes) 2024-09-16T13:18:27.213 INFO:teuthology.orchestra.run.smithi098.stdout:5395 writesame 0x7df8a7 thru 0x7e4505 data_size 0x39(0x4c5f bytes) 2024-09-16T13:18:27.213 INFO:teuthology.orchestra.run.smithi098.stdout:5396 read 0x58c45b3 thru 0x58d3df9 (0xf847 bytes) 2024-09-16T13:18:27.214 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.214 INFO:teuthology.orchestra.run.smithi098.stdout:5398 read 0x2bd9050 thru 0x2be17ec (0x879d bytes) 2024-09-16T13:18:27.214 INFO:teuthology.orchestra.run.smithi098.stdout:5399 read 0x6942a2f thru 0x694fddf (0xd3b1 bytes) 2024-09-16T13:18:27.214 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.214 INFO:teuthology.orchestra.run.smithi098.stdout:5402 read 0x1bc8f1 thru 0x1bf5e3 (0x2cf3 bytes) 2024-09-16T13:18:27.214 INFO:teuthology.orchestra.run.smithi098.stdout:5403 punch from 0x7d341e2 to 0x7d3a059, (0x5e77 bytes) 2024-09-16T13:18:27.215 INFO:teuthology.orchestra.run.smithi098.stdout:5405 write 0xa812c3e thru 0xa821f9f (0xf362 bytes) 2024-09-16T13:18:27.215 INFO:teuthology.orchestra.run.smithi098.stdout:5406 trunc from 0xa821fa0 to 0xc9d0c6b 2024-09-16T13:18:27.215 INFO:teuthology.orchestra.run.smithi098.stdout:5407 write 0xaa09742 thru 0xaa11df9 (0x86b8 bytes) 2024-09-16T13:18:27.215 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:27.207+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:27.225 INFO:teuthology.orchestra.run.smithi098.stdout:5412 write 0xdd25739 thru 0xdd2998c (0x4254 bytes) 2024-09-16T13:18:27.232 INFO:teuthology.orchestra.run.smithi098.stdout:5413 write 0x17acf2 thru 0x181dca (0x70d9 bytes) 2024-09-16T13:18:27.249 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.249 INFO:teuthology.orchestra.run.smithi098.stdout:5415 writesame 0x919aa0d thru 0x919c830 data_size 0x6(0x1e24 bytes) 2024-09-16T13:18:27.256 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.256 INFO:teuthology.orchestra.run.smithi098.stdout:5417 punch from 0x547bce7 to 0x5489eca, (0xe1e3 bytes) 2024-09-16T13:18:27.256 INFO:teuthology.orchestra.run.smithi098.stdout:5418 read 0xb6517ea thru 0xb65d8a6 (0xc0bd bytes) 2024-09-16T13:18:27.256 INFO:teuthology.orchestra.run.smithi098.stdout:5420 punch from 0x1167dc4 to 0x1177c95, (0xfed1 bytes) 2024-09-16T13:18:27.256 INFO:teuthology.orchestra.run.smithi098.stdout:5422 write 0x223911b thru 0x2245dc7 (0xccad bytes) 2024-09-16T13:18:27.264 INFO:teuthology.orchestra.run.smithi098.stdout:5424 read 0x5826ca4 thru 0x582b215 (0x4572 bytes) 2024-09-16T13:18:27.265 INFO:teuthology.orchestra.run.smithi098.stdout:5425 writesame 0xa263a8c thru 0xa270966 data_size 0x23(0xcedb bytes) 2024-09-16T13:18:27.274 INFO:teuthology.orchestra.run.smithi098.stdout:5426 punch from 0x57d5d75 to 0x57e20ad, (0xc338 bytes) 2024-09-16T13:18:27.274 INFO:teuthology.orchestra.run.smithi098.stdout:5427 write 0x3574c5e thru 0x3577ec2 (0x3265 bytes) 2024-09-16T13:18:27.280 INFO:teuthology.orchestra.run.smithi098.stdout:5428 read 0xbb7d45e thru 0xbb8b169 (0xdd0c bytes) 2024-09-16T13:18:27.280 INFO:teuthology.orchestra.run.smithi098.stdout:5429 read 0xd44a49c thru 0xd457d8b (0xd8f0 bytes) 2024-09-16T13:18:27.280 INFO:teuthology.orchestra.run.smithi098.stdout:5431 write 0xd0e0f83 thru 0xd0eb55b (0xa5d9 bytes) 2024-09-16T13:18:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:18:27.296 INFO:teuthology.orchestra.run.smithi098.stdout:5432 writesame 0x30506c1 thru 0x3057dea data_size 0x7(0x772a bytes) 2024-09-16T13:18:27.303 INFO:teuthology.orchestra.run.smithi098.stdout:5434 writesame 0xb9b6fc3 thru 0xb9c5c9a data_size 0x6a(0xecd8 bytes) 2024-09-16T13:18:27.313 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:27.307+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:27.333 INFO:teuthology.orchestra.run.smithi098.stdout:5435 trunc from 0xdd2998d to 0x2b49690 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:85503 3.2 3.dca49ce2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:27.335709+0000", 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.8087000000000002e-05, 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9475999999999997e-05, 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "queued for pg", 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.338 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 85503 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:27.335709+0000", 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:27.335709+0000", 2024-09-16T13:18:27.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:27.335710+0000", 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.096e-06 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:27.335716+0000", 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8069999999999996e-06 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:27.335731+0000", 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4737e-05 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:27.335737+0000", 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8799999999999996e-06 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 1 2024-09-16T13:18:27.341 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:27.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:27.370 INFO:teuthology.orchestra.run.smithi098.stdout:5436 trunc from 0x2b49690 to 0x7e4fb56 2024-09-16T13:18:27.395 INFO:teuthology.orchestra.run.smithi098.stdout:5439 write 0x851d91e thru 0x851e474 (0xb57 bytes) 2024-09-16T13:18:27.407 INFO:teuthology.orchestra.run.smithi098.stdout:5440 trunc from 0x851e475 to 0xa519584 2024-09-16T13:18:27.413 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:27.404+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:27.423 INFO:teuthology.orchestra.run.smithi098.stdout:5441 punch from 0x1188554 to 0x118b695, (0x3141 bytes) 2024-09-16T13:18:27.423 INFO:teuthology.orchestra.run.smithi098.stdout:5442 read 0x793e9ba thru 0x7943398 (0x49df bytes) 2024-09-16T13:18:27.423 INFO:teuthology.orchestra.run.smithi098.stdout:5443 read 0xe408f5 thru 0xe4a6f1 (0x9dfd bytes) 2024-09-16T13:18:27.423 INFO:teuthology.orchestra.run.smithi098.stdout:5444 punch from 0x9d5f452 to 0x9d67bdc, (0x878a bytes) 2024-09-16T13:18:27.423 INFO:teuthology.orchestra.run.smithi098.stdout:5445 read 0x2900d0a thru 0x290b86b (0xab62 bytes) 2024-09-16T13:18:27.423 INFO:teuthology.orchestra.run.smithi098.stdout:5446 read 0x5f022b1 thru 0x5f0e779 (0xc4c9 bytes) 2024-09-16T13:18:27.423 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.442 INFO:teuthology.orchestra.run.smithi098.stdout:5448 write 0xd7eee69 thru 0xd7faeb0 (0xc048 bytes) 2024-09-16T13:18:27.450 INFO:teuthology.orchestra.run.smithi098.stdout:5449 read 0x7cb6070 thru 0x7cb70f0 (0x1081 bytes) 2024-09-16T13:18:27.450 INFO:teuthology.orchestra.run.smithi098.stdout:5450 write 0xb3e00d4 thru 0xb3edbed (0xdb1a bytes) 2024-09-16T13:18:27.457 INFO:teuthology.orchestra.run.smithi098.stdout:5451 read 0x4d16b67 thru 0x4d1a6e0 (0x3b7a bytes) 2024-09-16T13:18:27.457 INFO:teuthology.orchestra.run.smithi098.stdout:5452 punch from 0xa7ccdfc to 0xa7cff9e, (0x31a2 bytes) 2024-09-16T13:18:27.457 INFO:teuthology.orchestra.run.smithi098.stdout:5453 read 0x5539783 thru 0x553c2d0 (0x2b4e bytes) 2024-09-16T13:18:27.472 INFO:teuthology.orchestra.run.smithi098.stdout:5454 write 0xe7184f6 thru 0xe71ea01 (0x650c bytes) 2024-09-16T13:18:27.474 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:27.474 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:27.474 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:27.474 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:27.474 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:27.478 INFO:teuthology.orchestra.run.smithi098.stdout:5455 read 0x316edd6 thru 0x3172801 (0x3a2c bytes) 2024-09-16T13:18:27.479 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.479 INFO:teuthology.orchestra.run.smithi098.stdout:5458 write 0x5eea57 thru 0x5f5a52 (0x6ffc bytes) 2024-09-16T13:18:27.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:27.495 INFO:teuthology.orchestra.run.smithi098.stdout:5459 write 0xe7d56df thru 0xe7d6ad5 (0x13f7 bytes) 2024-09-16T13:18:27.501 INFO:teuthology.orchestra.run.smithi098.stdout:5462 writesame 0x51144a thru 0x51882a data_size 0x11(0x73e1 bytes) 2024-09-16T13:18:27.507 INFO:teuthology.orchestra.run.smithi098.stdout:5463 read 0x521116b thru 0x521eb7b (0xda11 bytes) 2024-09-16T13:18:27.507 INFO:teuthology.orchestra.run.smithi098.stdout:5464 write 0xb4fd2e0 thru 0xb508b7f (0xb8a0 bytes) 2024-09-16T13:18:27.513 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:27.507+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:27.517 INFO:teuthology.orchestra.run.smithi098.stdout:5466 writesame 0x8527a29 thru 0x8534fef data_size 0x1(0xd5c7 bytes) 2024-09-16T13:18:27.561 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.561 INFO:teuthology.orchestra.run.smithi098.stdout:5468 read 0xe540383 thru 0xe54c4d4 (0xc152 bytes) 2024-09-16T13:18:27.561 INFO:teuthology.orchestra.run.smithi098.stdout:5469 write 0x44edea0 thru 0x44ef75f (0x18c0 bytes) 2024-09-16T13:18:27.568 INFO:teuthology.orchestra.run.smithi098.stdout:5470 read 0x7f2fb02 thru 0x7f379cf (0x7ece bytes) 2024-09-16T13:18:27.568 INFO:teuthology.orchestra.run.smithi098.stdout:5471 read 0x12a62dd thru 0x12ab546 (0x526a bytes) 2024-09-16T13:18:27.581 INFO:teuthology.orchestra.run.smithi098.stdout:5472 trunc from 0xe7d6ad6 to 0x7a879e9 2024-09-16T13:18:27.599 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:27.599 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:27.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:27.599 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:27.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.599 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62278 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.599 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:27.599 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278944199, 2024-09-16T13:18:27.599 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027907498999999999, 2024-09-16T13:18:27.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62278 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315829+0000", 2024-09-16T13:18:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.79e-06 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315835+0000", 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.682e-06 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315838+0000", 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0730000000000001e-06 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343640+0000", 2024-09-16T13:18:27.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027802384999999999 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343683+0000", 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3058e-05 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343734+0000", 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1104999999999999e-05 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.603 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.604 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.604 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:27.604 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278929637999999, 2024-09-16T13:18:27.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:18:27.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.604 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.604 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:18:27.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.608 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278905672, 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.609 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:18:27.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.613 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278890501999999, 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:18:27.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:18:27.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.616 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:18:27.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:18:27.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:27.618 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278876079, 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:18:27.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278854834000001, 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:18:27.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278840264999999, 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:18:27.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:18:27.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278821496000001, 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.632 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.636 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278806600999999, 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.637 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:18:27.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:18:27.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.27879239, 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.641 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.643 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.645 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278779021, 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:27.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:18:27.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:18:27.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278764339, 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.650 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:18:27.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:18:27.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:18:27.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:18:27.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:18:27.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278749169999999, 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:27.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278734529999999, 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.659 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:18:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278703802000001, 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:18:27.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278682785000001, 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.668 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:27.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:18:27.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278662677, 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:18:27.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:18:27.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:27.677 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278642774, 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:27.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:18:27.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278627561, 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:18:27.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 14.278612026999999, 2024-09-16T13:18:27.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:18:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:27.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:27.691 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:27.691 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:27.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:27.693 INFO:teuthology.orchestra.run.smithi098.stdout:5473 trunc from 0x7a879e9 to 0x5bed73f 2024-09-16T13:18:27.693 INFO:teuthology.orchestra.run.smithi098.stdout:5474 write 0x8617aea thru 0x862770f (0xfc26 bytes) 2024-09-16T13:18:27.693 INFO:teuthology.orchestra.run.smithi098.stdout:5475 read 0x6f3733 thru 0x6fde14 (0xa6e2 bytes) 2024-09-16T13:18:27.693 INFO:teuthology.orchestra.run.smithi098.stdout:5476 read 0x4fcae67 thru 0x4fce1bd (0x3357 bytes) 2024-09-16T13:18:27.693 INFO:teuthology.orchestra.run.smithi098.stdout:5477 write 0x8e311a6 thru 0x8e39f2d (0x8d88 bytes) 2024-09-16T13:18:27.694 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:27.683+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:27.695 INFO:teuthology.orchestra.run.smithi098.stdout:5479 read 0x58d622e thru 0x58dc2f0 (0x60c3 bytes) 2024-09-16T13:18:27.695 INFO:teuthology.orchestra.run.smithi098.stdout:5480 read 0x3fad567 thru 0x3fb57dd (0x8277 bytes) 2024-09-16T13:18:27.695 INFO:teuthology.orchestra.run.smithi098.stdout:5481 read 0x8bbcb32 thru 0x8bc4748 (0x7c17 bytes) 2024-09-16T13:18:27.695 INFO:teuthology.orchestra.run.smithi098.stdout:5482 read 0x15c14d6 thru 0x15c1e97 (0x9c2 bytes) 2024-09-16T13:18:27.695 INFO:teuthology.orchestra.run.smithi098.stdout:5483 writesame 0x72eb9d3 thru 0x72faafa data_size 0x4(0xf128 bytes) 2024-09-16T13:18:27.716 INFO:teuthology.orchestra.run.smithi098.stdout:5484 write 0x4abbfda thru 0x4aca3f4 (0xe41b bytes) 2024-09-16T13:18:27.725 INFO:teuthology.orchestra.run.smithi098.stdout:5485 read 0x2a058b thru 0x2a43de (0x3e54 bytes) 2024-09-16T13:18:27.726 INFO:teuthology.orchestra.run.smithi098.stdout:5487 read 0x3684ebf thru 0x3687720 (0x2862 bytes) 2024-09-16T13:18:27.726 INFO:teuthology.orchestra.run.smithi098.stdout:5489 read 0x45b173b thru 0x45b5b5b (0x4421 bytes) 2024-09-16T13:18:27.726 INFO:teuthology.orchestra.run.smithi098.stdout:5490 writesame 0x1cdec4f thru 0x1ce29ce data_size 0x18(0x3d80 bytes) 2024-09-16T13:18:27.743 INFO:teuthology.orchestra.run.smithi098.stdout:5492 write 0x94d789f thru 0x94e1bcf (0xa331 bytes) 2024-09-16T13:18:27.752 INFO:teuthology.orchestra.run.smithi098.stdout:5493 write 0x497f0a1 thru 0x49842ef (0x524f bytes) 2024-09-16T13:18:27.759 INFO:teuthology.orchestra.run.smithi098.stdout:5494 punch from 0x179281a to 0x179fe46, (0xd62c bytes) 2024-09-16T13:18:27.759 INFO:teuthology.orchestra.run.smithi098.stdout:5495 read 0x6386e01 thru 0x639271f (0xb91f bytes) 2024-09-16T13:18:27.759 INFO:teuthology.orchestra.run.smithi098.stdout:5496 read 0x6505c0d thru 0x6505d20 (0x114 bytes) 2024-09-16T13:18:27.760 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.760 INFO:teuthology.orchestra.run.smithi098.stdout:5498 read 0x11a746 thru 0x126a01 (0xc2bc bytes) 2024-09-16T13:18:27.760 INFO:teuthology.orchestra.run.smithi098.stdout:5501 read 0x7f23087 thru 0x7f29d71 (0x6ceb bytes) 2024-09-16T13:18:27.772 INFO:teuthology.orchestra.run.smithi098.stdout:5505 writesame 0x99fe718 thru 0x99ffe17 data_size 0x40(0x1700 bytes) 2024-09-16T13:18:27.775 INFO:teuthology.orchestra.run.smithi098.stdout:5507 trunc from 0x99ffe18 to 0xaf0f06d 2024-09-16T13:18:27.787 INFO:teuthology.orchestra.run.smithi098.stdout:5508 write 0x2accc33 thru 0x2ad88b1 (0xbc7f bytes) 2024-09-16T13:18:27.792 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:27.785+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:27.797 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.797 INFO:teuthology.orchestra.run.smithi098.stdout:5512 punch from 0x486608d to 0x486b4ba, (0x542d bytes) 2024-09-16T13:18:27.797 INFO:teuthology.orchestra.run.smithi098.stdout:5514 read 0x3bd9460 thru 0x3be2e24 (0x99c5 bytes) 2024-09-16T13:18:27.797 INFO:teuthology.orchestra.run.smithi098.stdout:5516 read 0x3b06452 thru 0x3b15eb6 (0xfa65 bytes) 2024-09-16T13:18:27.797 INFO:teuthology.orchestra.run.smithi098.stdout:5518 writesame 0x8b45b97 thru 0x8b4aaf6 data_size 0x28(0x4f60 bytes) 2024-09-16T13:18:27.805 INFO:teuthology.orchestra.run.smithi098.stdout:5519 write 0x3c328c6 thru 0x3c416e4 (0xee1f bytes) 2024-09-16T13:18:27.806 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:27.806 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:27.806 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:27.806 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:27.815 INFO:teuthology.orchestra.run.smithi098.stdout:5520 write 0xa2af091 thru 0xa2b204b (0x2fbb bytes) 2024-09-16T13:18:27.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:27.823 INFO:teuthology.orchestra.run.smithi098.stdout:5522 punch from 0x31994e1 to 0x3199770, (0x28f bytes) 2024-09-16T13:18:27.824 INFO:teuthology.orchestra.run.smithi098.stdout:5524 write 0x6e1633 thru 0x6e2ca7 (0x1675 bytes) 2024-09-16T13:18:27.826 INFO:teuthology.orchestra.run.smithi098.stdout:5525 read 0xab79404 thru 0xab86824 (0xd421 bytes) 2024-09-16T13:18:27.827 INFO:teuthology.orchestra.run.smithi098.stdout:5526 write 0x2701c21 thru 0x2708dc8 (0x71a8 bytes) 2024-09-16T13:18:27.830 INFO:teuthology.orchestra.run.smithi098.stdout:5528 punch from 0xa440bdb to 0xa44f416, (0xe83b bytes) 2024-09-16T13:18:27.830 INFO:teuthology.orchestra.run.smithi098.stdout:5529 read 0x6a16d6b thru 0x6a1e6dc (0x7972 bytes) 2024-09-16T13:18:27.831 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.831 INFO:teuthology.orchestra.run.smithi098.stdout:5532 writesame 0xa11b4c4 thru 0xa1258db data_size 0x59(0xa418 bytes) 2024-09-16T13:18:27.843 INFO:teuthology.orchestra.run.smithi098.stdout:5535 read 0x6c1b40b thru 0x6c1c91e (0x1514 bytes) 2024-09-16T13:18:27.843 INFO:teuthology.orchestra.run.smithi098.stdout:5536 read 0xa9e10dd thru 0xa9e78f2 (0x6816 bytes) 2024-09-16T13:18:27.844 INFO:teuthology.orchestra.run.smithi098.stdout:5537 punch from 0x83826ed to 0x8388184, (0x5a97 bytes) 2024-09-16T13:18:27.856 INFO:teuthology.orchestra.run.smithi098.stdout:5540 trunc from 0xaf0f06d to 0x5492bae 2024-09-16T13:18:27.880 INFO:teuthology.orchestra.run.smithi098.stdout:5541 write 0x41d2723 thru 0x41ddde2 (0xb6c0 bytes) 2024-09-16T13:18:27.883 INFO:teuthology.orchestra.run.smithi098.stdout:5542 read 0x16fffb4 thru 0x17093d7 (0x9424 bytes) 2024-09-16T13:18:27.884 INFO:teuthology.orchestra.run.smithi098.stdout:5543 read 0x42bf622 thru 0x42c202c (0x2a0b bytes) 2024-09-16T13:18:27.893 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:27.885+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:27.913 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:27.913 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:27.913 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:27.913 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:27.913 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:27.916 INFO:teuthology.orchestra.run.smithi098.stdout:5544 writesame 0xe85e9d8 thru 0xe869676 data_size 0x3b(0xac9f bytes) 2024-09-16T13:18:27.916 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.916 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:27.916 INFO:teuthology.orchestra.run.smithi098.stdout:5549 write 0x3a95fcb thru 0x3aa16f9 (0xb72f bytes) 2024-09-16T13:18:27.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:27.925 INFO:teuthology.orchestra.run.smithi098.stdout:5550 punch from 0xbfb5b7b to 0xbfc3267, (0xd6ec bytes) 2024-09-16T13:18:27.925 INFO:teuthology.orchestra.run.smithi098.stdout:5553 punch from 0x595da5f to 0x5969461, (0xba02 bytes) 2024-09-16T13:18:27.950 INFO:teuthology.orchestra.run.smithi098.stdout:5555 trunc from 0xe869677 to 0x15fc33c 2024-09-16T13:18:27.994 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:27.987+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:27.998 INFO:teuthology.orchestra.run.smithi098.stdout:5556 write 0x64196b6 thru 0x6419c62 (0x5ad bytes) 2024-09-16T13:18:28.017 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.802765173999999, 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:18:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75320 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1114112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5575254479999998, 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048391086999999999, 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75320 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455024+0000", 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455026+0000", 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:18:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455033+0000", 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8630000000000002e-06 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455037+0000", 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.608e-06 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503178+0000", 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048141646000000003 2024-09-16T13:18:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503284+0000", 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105917 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503415+0000", 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130359 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75321 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1441792~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5575084560000008, 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048644984000000002, 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75321 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455041+0000", 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-16T13:18:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455043+0000", 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0480000000000001e-06 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455048+0000", 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455052+0000", 2024-09-16T13:18:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000002e-06 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503461+0000", 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048409079000000001 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503580+0000", 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119234 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503685+0000", 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105006 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75322 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 2064384~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5574830589999991, 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048835044000000001, 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75322 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2599999999999998e-07 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455068+0000", 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.092e-06 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455074+0000", 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.925e-06 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455078+0000", 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503740+0000", 2024-09-16T13:18:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048661514000000003 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503822+0000", 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2196999999999999e-05 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503901+0000", 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9091000000000006e-05 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75344 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 589824~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5569794189999993, 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048412121000000002, 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75344 2024-09-16T13:18:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455570+0000", 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455572+0000", 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2749999999999998e-06 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455578+0000", 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455582+0000", 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4789999999999998e-06 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503719+0000", 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048137054999999998 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503819+0000", 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9938000000000004e-05 2024-09-16T13:18:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503982+0000", 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016225500000000001 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75345 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 917504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5569627530000005, 2024-09-16T13:18:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048702503000000001, 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75345 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455587+0000", 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455589+0000", 2024-09-16T13:18:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:18:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455594+0000", 2024-09-16T13:18:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6840000000000001e-06 2024-09-16T13:18:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455598+0000", 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7009999999999998e-06 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504027+0000", 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048429382999999999 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504123+0000", 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5251e-05 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504289+0000", 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016599200000000001 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75346 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1212416~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.556947396, 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048896186000000001, 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75346 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455602+0000", 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2799999999999996e-07 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455604+0000", 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455609+0000", 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455615+0000", 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2020000000000003e-06 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504315+0000", 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048700565000000001 2024-09-16T13:18:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504382+0000", 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7088999999999994e-05 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504498+0000", 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115408 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75347 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1540096~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5569288500000003, 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049121102999999999, 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75347 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7100000000000002e-07 2024-09-16T13:18:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455623+0000", 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455629+0000", 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455633+0000", 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9400000000000004e-06 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504546+0000", 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048912718000000001 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504623+0000", 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7726999999999996e-05 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504741+0000", 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117874 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75353 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 98304~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5567931060000006, 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048929180000000003, 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75353 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455758+0000", 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8789999999999999e-06 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455763+0000", 2024-09-16T13:18:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2190000000000003e-06 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455767+0000", 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504529+0000", 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048761566999999999 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504587+0000", 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8564999999999999e-05 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504685+0000", 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7754e-05 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75354 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 393216~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5567580359999997, 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049086520000000002, 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75354 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455794+0000", 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.508e-06 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455801+0000", 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5630000000000003e-06 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455805+0000", 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504704+0000", 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048898981000000001 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504759+0000", 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4330000000000003e-05 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504877+0000", 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118608 2024-09-16T13:18:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75355 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.556739404, 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118086999999998, 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75355 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455810+0000", 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455812+0000", 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455818+0000", 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455821+0000", 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504914+0000", 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049092961999999997 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504928+0000", 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3210000000000001e-05 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75356 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5567236429999998, 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049129843999999999, 2024-09-16T13:18:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75356 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455826+0000", 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0399999999999996e-07 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455828+0000", 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-06 2024-09-16T13:18:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455835+0000", 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8410000000000002e-06 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455839+0000", 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2400000000000001e-06 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504947+0000", 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049108546000000003 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504955+0000", 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6109999999999996e-06 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75357 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5567063789999995, 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049141246, 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75357 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455843+0000", 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999998e-07 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455845+0000", 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999998e-06 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455852+0000", 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455856+0000", 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6059999999999999e-06 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504974+0000", 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118385000000001 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504984+0000", 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7880000000000001e-06 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75358 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5566894300000005, 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049149627000000001, 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75358 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455860+0000", 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455862+0000", 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455867+0000", 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.395e-06 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455873+0000", 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0120000000000002e-06 2024-09-16T13:18:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505002+0000", 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049128988999999998 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505009+0000", 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7950000000000002e-06 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75370 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5566716770000006, 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049174830000000003, 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75370 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455878+0000", 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:18:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455880+0000", 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455885+0000", 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455889+0000", 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6090000000000001e-06 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505040+0000", 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049151242999999997 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505052+0000", 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2055999999999999e-05 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75371 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5566565410000006, 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049181864999999998, 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75371 2024-09-16T13:18:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455893+0000", 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455895+0000", 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455900+0000", 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.536e-06 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455904+0000", 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505067+0000", 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049163224999999998 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505074+0000", 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:18:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75372 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5566414030000004, 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049214870000000001, 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75372 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455908+0000", 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.89e-07 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455910+0000", 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455915+0000", 2024-09-16T13:18:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455919+0000", 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.472e-06 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505109+0000", 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049190747 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505122+0000", 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3009e-05 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75373 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5566268860000001, 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049224933999999998, 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75373 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455924+0000", 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455930+0000", 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4929999999999998e-06 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455936+0000", 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4459999999999998e-06 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505140+0000", 2024-09-16T13:18:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049203510999999998 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505147+0000", 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1670000000000002e-06 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75374 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5566088069999999, 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049235427999999998, 2024-09-16T13:18:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75374 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455946+0000", 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860000000000001e-06 2024-09-16T13:18:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455953+0000", 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5119999999999997e-06 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455956+0000", 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.579e-06 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505166+0000", 2024-09-16T13:18:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049209527000000003 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505175+0000", 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75375 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:28.101 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.5565885270000006, 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049243375999999998, 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75375 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455961+0000", 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455963+0000", 2024-09-16T13:18:28.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729999999999999e-06 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455970+0000", 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2520000000000002e-06 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455975+0000", 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505198+0000", 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049222852999999997 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505204+0000", 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.105 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:28.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:28.108 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:28.099+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:28.108 INFO:teuthology.orchestra.run.smithi098.stdout:5557 writesame 0xcdd99db thru 0xcde6753 data_size 0x1(0xcd79 bytes) 2024-09-16T13:18:28.108 INFO:teuthology.orchestra.run.smithi098.stdout:5558 write 0x73a751b thru 0x73b1378 (0x9e5e bytes) 2024-09-16T13:18:28.109 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:28.109 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:28.109 INFO:teuthology.orchestra.run.smithi098.stdout:5562 writesame 0x7f0bb12 thru 0x7f13973 data_size 0x1(0x7e62 bytes) 2024-09-16T13:18:28.109 INFO:teuthology.orchestra.run.smithi098.stdout:5563 writesame 0x17d03d3 thru 0x17d973a data_size 0x6a(0x9368 bytes) 2024-09-16T13:18:28.109 INFO:teuthology.orchestra.run.smithi098.stdout:5564 write 0xa8dfeca thru 0xa8e281c (0x2953 bytes) 2024-09-16T13:18:28.109 INFO:teuthology.orchestra.run.smithi098.stdout:5565 write 0x733a464 thru 0x733d10a (0x2ca7 bytes) 2024-09-16T13:18:28.109 INFO:teuthology.orchestra.run.smithi098.stdout:5566 read 0xdaa512 thru 0xdb6edd (0xc9cc bytes) 2024-09-16T13:18:28.109 INFO:teuthology.orchestra.run.smithi098.stdout:5567 trunc from 0xcde6754 to 0xae353b9 2024-09-16T13:18:28.112 INFO:teuthology.orchestra.run.smithi098.stdout:5568 writesame 0x3d67532 thru 0x3d6e415 data_size 0x2f(0x6ee4 bytes) 2024-09-16T13:18:28.120 INFO:teuthology.orchestra.run.smithi098.stdout:5570 read 0x9306874 thru 0x93150d7 (0xe864 bytes) 2024-09-16T13:18:28.140 INFO:teuthology.orchestra.run.smithi098.stdout:5572 write 0xd3d0fc3 thru 0xd3d25f5 (0x1633 bytes) 2024-09-16T13:18:28.154 INFO:teuthology.orchestra.run.smithi098.stdout:5573 trunc from 0xd3d25f6 to 0x976e792 2024-09-16T13:18:28.180 INFO:teuthology.orchestra.run.smithi098.stdout:5574 read 0x6634084 thru 0x6637ab5 (0x3a32 bytes) 2024-09-16T13:18:28.180 INFO:teuthology.orchestra.run.smithi098.stdout:5575 punch from 0x77149dd to 0x7717047, (0x266a bytes) 2024-09-16T13:18:28.195 INFO:teuthology.orchestra.run.smithi098.stdout:5576 write 0xae68cf2 thru 0xae6995d (0xc6c bytes) 2024-09-16T13:18:28.201 INFO:teuthology.orchestra.run.smithi098.stdout:5578 write 0x44361ae thru 0x443a7b8 (0x460b bytes) 2024-09-16T13:18:28.207 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:28.201+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:28.209 INFO:teuthology.orchestra.run.smithi098.stdout:5580 write 0x80cf02d thru 0x80d63ce (0x73a2 bytes) 2024-09-16T13:18:28.218 INFO:teuthology.orchestra.run.smithi098.stdout:5581 writesame 0x641a57c thru 0x64270d4 data_size 0x1(0xcb59 bytes) 2024-09-16T13:18:28.262 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:28.263 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:28.263 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:28.263 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:28.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:28.273 INFO:teuthology.orchestra.run.smithi098.stdout:5582 read 0x3c0239a thru 0x3c11db8 (0xfa1f bytes) 2024-09-16T13:18:28.275 INFO:teuthology.orchestra.run.smithi098.stdout:5583 write 0x498ff68 thru 0x499ec05 (0xec9e bytes) 2024-09-16T13:18:28.281 INFO:teuthology.orchestra.run.smithi098.stdout:5585 punch from 0xa3cbbde to 0xa3dafc3, (0xf3e5 bytes) 2024-09-16T13:18:28.281 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:28.295 INFO:teuthology.orchestra.run.smithi098.stdout:5587 writesame 0xcb39ccd thru 0xcb3f8fe data_size 0x1(0x5c32 bytes) 2024-09-16T13:18:28.295 INFO:teuthology.orchestra.run.smithi098.stdout:5588 write 0x83d6c3b thru 0x83dfe70 (0x9236 bytes) 2024-09-16T13:18:28.300 INFO:teuthology.orchestra.run.smithi098.stdout:5589 read 0xafe9df0 thru 0xaff652e (0xc73f bytes) 2024-09-16T13:18:28.300 INFO:teuthology.orchestra.run.smithi098.stdout:5590 punch from 0x44edc84 to 0x44fa15b, (0xc4d7 bytes) 2024-09-16T13:18:28.300 INFO:teuthology.orchestra.run.smithi098.stdout:5591 read 0x8a4c931 thru 0x8a5496b (0x803b bytes) 2024-09-16T13:18:28.300 INFO:teuthology.orchestra.run.smithi098.stdout:5592 write 0xc57ba23 thru 0xc587e3c (0xc41a bytes) 2024-09-16T13:18:28.308 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:28.303+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:28.310 INFO:teuthology.orchestra.run.smithi098.stdout:5594 read 0x311788d thru 0x31210e4 (0x9858 bytes) 2024-09-16T13:18:28.310 INFO:teuthology.orchestra.run.smithi098.stdout:5595 read 0x47edff7 thru 0x47f8cd7 (0xace1 bytes) 2024-09-16T13:18:28.311 INFO:teuthology.orchestra.run.smithi098.stdout:5596 writesame 0xab167ae thru 0xab22353 data_size 0x1(0xbba6 bytes) 2024-09-16T13:18:28.354 INFO:teuthology.orchestra.run.smithi098.stdout:5597 write 0x67855af thru 0x678ce41 (0x7893 bytes) 2024-09-16T13:18:28.370 INFO:teuthology.orchestra.run.smithi098.stdout:5598 trunc from 0xcb3f8ff to 0x813d143 2024-09-16T13:18:28.372 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:28.372 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:28.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:28.397 INFO:teuthology.orchestra.run.smithi098.stdout:5599 punch from 0x67b252b to 0x67befed, (0xcac2 bytes) 2024-09-16T13:18:28.397 INFO:teuthology.orchestra.run.smithi098.stdout:5600 write 0x75b30fa thru 0x75c2e44 (0xfd4b bytes) 2024-09-16T13:18:28.409 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:28.401+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:28.430 INFO:teuthology.orchestra.run.smithi098.stdout:5601 writesame 0xec86bd8 thru 0xec8d240 data_size 0x9(0x6669 bytes) 2024-09-16T13:18:28.430 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:28.430 INFO:teuthology.orchestra.run.smithi098.stdout:5603 writesame 0x5b64e70 thru 0x5b70fe0 data_size 0x3(0xc171 bytes) 2024-09-16T13:18:28.452 INFO:teuthology.orchestra.run.smithi098.stdout:5604 read 0x5ba8484 thru 0x5baff11 (0x7a8e bytes) 2024-09-16T13:18:28.452 INFO:teuthology.orchestra.run.smithi098.stdout:5605 read 0x32ce9ed thru 0x32da08c (0xb6a0 bytes) 2024-09-16T13:18:28.453 INFO:teuthology.orchestra.run.smithi098.stdout:5606 write 0x2cf38af thru 0x2cfbdda (0x852c bytes) 2024-09-16T13:18:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:5607 read 0x26c52b1 thru 0x26c7d65 (0x2ab5 bytes) 2024-09-16T13:18:28.463 INFO:teuthology.orchestra.run.smithi098.stdout:5608 trunc from 0xec8d241 to 0xd7bf9fc 2024-09-16T13:18:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:5609 writesame 0xd9ac9a5 thru 0xd9ba918 data_size 0x3f(0xdf74 bytes) 2024-09-16T13:18:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:28.487 INFO:teuthology.orchestra.run.smithi098.stdout:5612 write 0xd223b13 thru 0xd22d013 (0x9501 bytes) 2024-09-16T13:18:28.493 INFO:teuthology.orchestra.run.smithi098.stdout:5617 write 0x8d941ef thru 0x8d9a8aa (0x66bc bytes) 2024-09-16T13:18:28.511 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:28.503+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:28.525 INFO:teuthology.orchestra.run.smithi098.stdout:5618 trunc from 0xd9ba919 to 0x2029c82 2024-09-16T13:18:28.534 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.704925521, 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.536 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:18:28.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.536 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:28.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:18:28.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:18:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:18:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:18:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:18:28.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:28.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:18:28.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:18:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:18:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:18:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:18:28.542 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82800 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1130496~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6688674260000003, 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050353660000000001, 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82800 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858530+0000", 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858534+0000", 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8229999999999997e-06 2024-09-16T13:18:28.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858547+0000", 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3302000000000001e-05 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858557+0000", 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-05 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908637+0000", 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050079811000000002 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908731+0000", 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4085000000000005e-05 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908883+0000", 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000151676 2024-09-16T13:18:28.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82801 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1212416~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6688315310000004, 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050667642999999998, 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82801 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:28.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858566+0000", 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3499999999999995e-07 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858570+0000", 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8299999999999998e-06 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858590+0000", 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9831e-05 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858597+0000", 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6729999999999992e-06 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908930+0000", 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050332492999999999 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909036+0000", 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010603999999999999 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909233+0000", 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019704100000000001 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.552 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82802 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6687912350000005, 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050929965000000001, 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82802 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858610+0000", 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0820000000000001e-06 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858655+0000", 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4901000000000003e-05 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858668+0000", 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2343e-05 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909278+0000", 2024-09-16T13:18:28.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050610005 2024-09-16T13:18:28.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909374+0000", 2024-09-16T13:18:28.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6399999999999999e-05 2024-09-16T13:18:28.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909535+0000", 2024-09-16T13:18:28.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016137599999999999 2024-09-16T13:18:28.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.557 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82803 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1474560~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6686823840000002, 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051051886999999997, 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82803 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858715+0000", 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999995e-07 2024-09-16T13:18:28.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858719+0000", 2024-09-16T13:18:28.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.732e-06 2024-09-16T13:18:28.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858728+0000", 2024-09-16T13:18:28.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6730000000000004e-06 2024-09-16T13:18:28.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858733+0000", 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.532e-06 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909572+0000", 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050839399 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909647+0000", 2024-09-16T13:18:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5073000000000004e-05 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909766+0000", 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118805 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82804 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1654784~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:28.562 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6686580119999999, 2024-09-16T13:18:28.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051320085000000001, 2024-09-16T13:18:28.563 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.563 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.563 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82804 2024-09-16T13:18:28.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.563 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:28.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3399999999999998e-07 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858742+0000", 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4150000000000002e-06 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858748+0000", 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0090000000000004e-06 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858753+0000", 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1649999999999997e-06 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909812+0000", 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051058612000000003 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909907+0000", 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5795e-05 2024-09-16T13:18:28.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910059+0000", 2024-09-16T13:18:28.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015135499999999999 2024-09-16T13:18:28.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.567 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82805 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1736704~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6686380960000005, 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051834002999999997, 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82805 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:18:28.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858762+0000", 2024-09-16T13:18:28.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4959999999999999e-06 2024-09-16T13:18:28.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858768+0000", 2024-09-16T13:18:28.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8830000000000003e-06 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858771+0000", 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9319999999999998e-06 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910078+0000", 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051306696999999998 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910163+0000", 2024-09-16T13:18:28.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5065e-05 2024-09-16T13:18:28.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910593+0000", 2024-09-16T13:18:28.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042940799999999999 2024-09-16T13:18:28.572 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.572 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82806 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1916928~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6686213150000002, 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052120149999999997, 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.573 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82806 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858776+0000", 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2600000000000002e-07 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858778+0000", 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3070000000000001e-06 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858784+0000", 2024-09-16T13:18:28.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858788+0000", 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8870000000000002e-06 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910643+0000", 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051854355999999997 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910735+0000", 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2454000000000006e-05 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910896+0000", 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016055600000000001 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82807 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1998848~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:28.577 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6686021469999996, 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052345503000000002, 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82807 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:28.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858797+0000", 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858804+0000", 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0079999999999996e-06 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858808+0000", 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910935+0000", 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052126695000000001 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911010+0000", 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4764000000000004e-05 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911140+0000", 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013022500000000001 2024-09-16T13:18:28.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82830 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 770048~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6685841369999999, 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052616234999999997, 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82830 2024-09-16T13:18:28.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:28.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858815+0000", 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3130000000000001e-06 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858821+0000", 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7200000000000003e-06 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858838+0000", 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6492000000000001e-05 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911178+0000", 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052340684999999998 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911272+0000", 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4106999999999999e-05 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911429+0000", 2024-09-16T13:18:28.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156389 2024-09-16T13:18:28.586 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.586 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.586 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.586 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82831 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.587 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:28.587 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6685503500000003, 2024-09-16T13:18:28.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052803142999999997, 2024-09-16T13:18:28.587 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.587 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.587 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82831 2024-09-16T13:18:28.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858848+0000", 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.209e-06 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858858+0000", 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0111e-05 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858870+0000", 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1976e-05 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858876+0000", 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5649999999999998e-06 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911449+0000", 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052573110999999999 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911521+0000", 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1209000000000002e-05 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911650+0000", 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012896199999999999 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82832 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1032192~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6685126889999999, 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053031663999999999, 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82832 2024-09-16T13:18:28.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858885+0000", 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6899999999999996e-07 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858889+0000", 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.087e-06 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858905+0000", 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5744999999999999e-05 2024-09-16T13:18:28.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858909+0000", 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7269999999999998e-06 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911673+0000", 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052763697999999998 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911767+0000", 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4054000000000004e-05 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911916+0000", 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148584 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82833 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1114112~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6684795860000001, 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053225354000000002, 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82833 2024-09-16T13:18:28.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858918+0000", 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858922+0000", 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.371e-06 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858933+0000", 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1076e-05 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858940+0000", 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9840000000000004e-06 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911944+0000", 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053004044 2024-09-16T13:18:28.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912021+0000", 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7189999999999998e-05 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912143+0000", 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012160200000000001 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82834 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1294336~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6684498340000005, 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053469475000000002, 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82834 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858948+0000", 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:28.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858952+0000", 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858970+0000", 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.889e-05 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858978+0000", 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912172+0000", 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053193670999999998 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912268+0000", 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6100000000000005e-05 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912416+0000", 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014817500000000001 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82835 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1376256~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6684095399999999, 2024-09-16T13:18:28.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053671286999999998, 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82835 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858988+0000", 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858992+0000", 2024-09-16T13:18:28.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:18:28.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858999+0000", 2024-09-16T13:18:28.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-16T13:18:28.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859003+0000", 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912456+0000", 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053453374999999997 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912526+0000", 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9237000000000001e-05 2024-09-16T13:18:28.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912658+0000", 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013284000000000001 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82836 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1556480~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.611 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.668389693, 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053937880000000001, 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82836 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859008+0000", 2024-09-16T13:18:28.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859010+0000", 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1950000000000002e-06 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859016+0000", 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8429999999999998e-06 2024-09-16T13:18:28.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859019+0000", 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912697+0000", 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053677322 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912796+0000", 2024-09-16T13:18:28.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8991000000000001e-05 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912945+0000", 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149207 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82837 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1638400~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:28.616 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6683736700000003, 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054154068999999999, 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82837 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859024+0000", 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859026+0000", 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859031+0000", 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6690000000000003e-06 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859035+0000", 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6239999999999999e-06 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912981+0000", 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053946409000000001 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913054+0000", 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3052999999999993e-05 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913177+0000", 2024-09-16T13:18:28.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122803 2024-09-16T13:18:28.621 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.621 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.621 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.621 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82838 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1818624~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.621 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:28.621 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6683582320000001, 2024-09-16T13:18:28.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054416554999999998, 2024-09-16T13:18:28.622 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.622 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.622 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82838 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859041+0000", 2024-09-16T13:18:28.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2139999999999999e-06 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859047+0000", 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6359999999999999e-06 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859051+0000", 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6519999999999999e-06 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913215+0000", 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054164344000000003 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913307+0000", 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2546999999999993e-05 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913455+0000", 2024-09-16T13:18:28.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147639 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82839 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1900544~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6683425700000001, 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054634963000000002, 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.627 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82839 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859055+0000", 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859057+0000", 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3910000000000001e-06 2024-09-16T13:18:28.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859066+0000", 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5150000000000006e-06 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859072+0000", 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5810000000000002e-06 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913493+0000", 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054420751000000003 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913563+0000", 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9938999999999996e-05 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913689+0000", 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012626299999999999 2024-09-16T13:18:28.631 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82840 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 2080768~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.6683155339999995, 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054884993999999999, 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:28.632 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82840 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:28.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859082+0000", 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5799999999999998e-07 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859086+0000", 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7510000000000002e-06 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859097+0000", 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1085e-05 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859104+0000", 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-06 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913718+0000", 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054614085 2024-09-16T13:18:28.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913814+0000", 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5514999999999996e-05 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913966+0000", 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015253 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:28.637 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:28.637 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:28.637 INFO:teuthology.orchestra.run.smithi098.stdout:5621 clone 38 order 22 su 524288 sc 15 2024-09-16T13:18:28.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:28.639 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:28.631+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:28.739 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:28.731+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:28.790 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:28.790 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:28.790 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:28.790 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:28.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:28.840 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:28.837+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:28.906 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:28.906 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:28.906 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:28.906 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:28.906 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:28.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:28.941 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:28.935+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:29.018 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:29.018 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:29.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:29.019 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:29.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.019 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.019 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:29.019 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132829057999999, 2024-09-16T13:18:29.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:18:29.019 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.019 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.019 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:29.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:18:29.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:18:29.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:18:29.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:18:29.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132810223, 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:18:29.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:29.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132791084000001, 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:18:29.028 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:29.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:18:29.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:29.033 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132774026, 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:18:29.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:18:29.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132756665999999, 2024-09-16T13:18:29.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:18:29.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:18:29.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132738957000001, 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.044 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:18:29.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132721658000001, 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:18:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.13268656, 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.053 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132663404999999, 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.057 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:18:29.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132622386000001, 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.062 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.063 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:18:29.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.063 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:18:29.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:18:29.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:18:29.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:18:29.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:18:29.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.067 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.068 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.068 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:29.068 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132599486, 2024-09-16T13:18:29.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:18:29.068 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.068 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.068 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.068 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:18:29.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:18:29.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:18:29.072 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132574436999999, 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:18:29.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:18:29.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:18:29.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:18:29.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:18:29.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132549216000001, 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:18:29.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:18:29.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:18:29.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132530850999999, 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:18:29.083 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:18:29.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:18:29.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:18:29.088 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.088 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.088 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.088 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.088 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132267714000001, 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:18:29.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:18:29.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:18:29.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:18:29.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:18:29.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132242706, 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:18:29.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:18:29.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.132214374, 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:18:29.098 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:18:29.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:18:29.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:18:29.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:18:29.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:18:29.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.103 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.131925676000002, 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:18:29.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:29.108 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.131905566, 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:18:29.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.127704642000001, 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:29.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:18:29.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.117 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.117 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.117 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:29.117 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:29.117 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:29.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:29.119 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:29.113+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:29.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:29.220 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:29.211+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:29.225 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:29.226 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:29.226 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:29.226 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:29.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:29.255 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:29.274 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:29.296 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:29.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:18:29.314 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:29.320 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:29.313+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:29.333 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:29.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:29.356 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:29.385 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:29.386 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:29.386 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:29.386 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:29.386 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:29.386 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:29.386 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:29.386 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:29.387 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:29.390 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:29.420 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:29.415+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:29.441 INFO:teuthology.orchestra.run.smithi098.stdout:truncating image image_client.0-clone37 from 0x2029c82 (overlap 0x15fc33c) to 0x8cb47e 2024-09-16T13:18:29.469 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:29.470 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:29.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:29.521 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:29.515+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:29.579 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:29.579 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:29.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:29.622 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:29.615+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:29.626 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #36, image image_client.0-clone36 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent37 2024-09-16T13:18:29.630 INFO:teuthology.orchestra.run.smithi098.stdout:5624 trunc from 0x2029c82 to 0x292981e 2024-09-16T13:18:29.648 INFO:teuthology.orchestra.run.smithi098.stdout:5625 write 0x9b825f thru 0x9c4be5 (0xc987 bytes) 2024-09-16T13:18:29.689 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:29.689 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:29.690 INFO:teuthology.orchestra.run.smithi098.stdout:5626 write 0xe23f194 thru 0xe2497f0 (0xa65d bytes) 2024-09-16T13:18:29.696 INFO:teuthology.orchestra.run.smithi098.stdout:5627 writesame 0xe106973 thru 0xe1144c8 data_size 0x19(0xdb56 bytes) 2024-09-16T13:18:29.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:29.705 INFO:teuthology.orchestra.run.smithi098.stdout:5628 read 0xd490300 thru 0xd49f762 (0xf463 bytes) 2024-09-16T13:18:29.705 INFO:teuthology.orchestra.run.smithi098.stdout:5630 read 0xad71add thru 0xad7b498 (0x99bc bytes) 2024-09-16T13:18:29.705 INFO:teuthology.orchestra.run.smithi098.stdout:5631 write 0x8947d7e thru 0x8952121 (0xa3a4 bytes) 2024-09-16T13:18:29.711 INFO:teuthology.orchestra.run.smithi098.stdout:5633 punch from 0x6a33f2d to 0x6a34f26, (0xff9 bytes) 2024-09-16T13:18:29.720 INFO:teuthology.orchestra.run.smithi098.stdout:5634 trunc from 0xe2497f1 to 0x998d85c 2024-09-16T13:18:29.723 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:29.715+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:29.745 INFO:teuthology.orchestra.run.smithi098.stdout:5635 write 0x92bb723 thru 0x92c79cd (0xc2ab bytes) 2024-09-16T13:18:29.761 INFO:teuthology.orchestra.run.smithi098.stdout:5636 trunc from 0x998d85c to 0x46910e9 2024-09-16T13:18:29.794 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:29.795 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:29.796 INFO:teuthology.orchestra.run.smithi098.stdout:5637 write 0x6797f04 thru 0x67a6c52 (0xed4f bytes) 2024-09-16T13:18:29.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:29.808 INFO:teuthology.orchestra.run.smithi098.stdout:5638 trunc from 0x67a6c53 to 0x8b8a179 2024-09-16T13:18:29.825 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:29.817+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:29.839 INFO:teuthology.orchestra.run.smithi098.stdout:5639 write 0xcac75e9 thru 0xcad6dbd (0xf7d5 bytes) 2024-09-16T13:18:29.844 INFO:teuthology.orchestra.run.smithi098.stdout:5640 read 0x60f2bc8 thru 0x60fe2b9 (0xb6f2 bytes) 2024-09-16T13:18:29.845 INFO:teuthology.orchestra.run.smithi098.stdout:5641 writesame 0x335876a thru 0x336018a data_size 0xd(0x7a21 bytes) 2024-09-16T13:18:29.858 INFO:teuthology.orchestra.run.smithi098.stdout:5642 write 0x4a10b19 thru 0x4a1ce86 (0xc36e bytes) 2024-09-16T13:18:29.864 INFO:teuthology.orchestra.run.smithi098.stdout:5643 read 0x8a11f51 thru 0x8a21237 (0xf2e7 bytes) 2024-09-16T13:18:29.881 INFO:teuthology.orchestra.run.smithi098.stdout:5645 trunc from 0xcad6dbe to 0x2ccf048 2024-09-16T13:18:29.912 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:29.912 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:29.926 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:29.917+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:29.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:29.935 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:29.935 INFO:teuthology.orchestra.run.smithi098.stdout:5652 read 0x1db99bf thru 0x1dba82a (0xe6c bytes) 2024-09-16T13:18:29.935 INFO:teuthology.orchestra.run.smithi098.stdout:5653 read 0x2acce76 thru 0x2acead5 (0x1c60 bytes) 2024-09-16T13:18:29.936 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:5655 trunc from 0x2ccf048 to 0x4a45378 2024-09-16T13:18:29.947 INFO:teuthology.orchestra.run.smithi098.stdout:5656 punch from 0x18da4a to 0x19cc9e, (0xf254 bytes) 2024-09-16T13:18:29.947 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:29.959 INFO:teuthology.orchestra.run.smithi098.stdout:5658 trunc from 0x4a45378 to 0xbdceb65 2024-09-16T13:18:29.970 INFO:teuthology.orchestra.run.smithi098.stdout:5659 writesame 0xb4a2cfa thru 0xb4ac1f0 data_size 0x5(0x94f7 bytes) 2024-09-16T13:18:29.984 INFO:teuthology.orchestra.run.smithi098.stdout:5660 write 0x9f880b5 thru 0x9f91386 (0x92d2 bytes) 2024-09-16T13:18:29.990 INFO:teuthology.orchestra.run.smithi098.stdout:5663 read 0x40adc57 thru 0x40af915 (0x1cbf bytes) 2024-09-16T13:18:29.990 INFO:teuthology.orchestra.run.smithi098.stdout:5664 read 0xb8b9428 thru 0xb8c7618 (0xe1f1 bytes) 2024-09-16T13:18:29.990 INFO:teuthology.orchestra.run.smithi098.stdout:5665 write 0x20a39c9 thru 0x20ae0d2 (0xa70a bytes) 2024-09-16T13:18:30.012 INFO:teuthology.orchestra.run.smithi098.stdout:5669 write 0xeba906f thru 0xebaeafe (0x5a90 bytes) 2024-09-16T13:18:30.019 INFO:teuthology.orchestra.run.smithi098.stdout:5670 writesame 0x68dd794 thru 0x68ed0b9 data_size 0x1(0xf926 bytes) 2024-09-16T13:18:30.027 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:30.019+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:30.035 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:30.035 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:30.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:30.078 INFO:teuthology.orchestra.run.smithi098.stdout:5672 writesame 0xe403191 thru 0xe403ee3 data_size 0x3(0xd53 bytes) 2024-09-16T13:18:30.084 INFO:teuthology.orchestra.run.smithi098.stdout:5673 punch from 0x979f03e to 0x97a70d8, (0x809a bytes) 2024-09-16T13:18:30.084 INFO:teuthology.orchestra.run.smithi098.stdout:5674 write 0x8c905d9 thru 0x8c925ff (0x2027 bytes) 2024-09-16T13:18:30.091 INFO:teuthology.orchestra.run.smithi098.stdout:5675 punch from 0x9b15089 to 0x9b21f17, (0xce8e bytes) 2024-09-16T13:18:30.091 INFO:teuthology.orchestra.run.smithi098.stdout:5676 punch from 0x6d96f9e to 0x6d9b412, (0x4474 bytes) 2024-09-16T13:18:30.091 INFO:teuthology.orchestra.run.smithi098.stdout:5678 write 0x55139cb thru 0x551e19a (0xa7d0 bytes) 2024-09-16T13:18:30.096 INFO:teuthology.orchestra.run.smithi098.stdout:5679 write 0xe025137 thru 0xe027b47 (0x2a11 bytes) 2024-09-16T13:18:30.101 INFO:teuthology.orchestra.run.smithi098.stdout:5681 write 0xd5fd821 thru 0xd5fe98d (0x116d bytes) 2024-09-16T13:18:30.106 INFO:teuthology.orchestra.run.smithi098.stdout:5682 writesame 0x3113f2e thru 0x311d106 data_size 0x1(0x91d9 bytes) 2024-09-16T13:18:30.128 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:30.121+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:30.137 INFO:teuthology.orchestra.run.smithi098.stdout:5683 writesame 0x81d48ab thru 0x81e2bb8 data_size 0x1(0xe30e bytes) 2024-09-16T13:18:30.179 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:30.179 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:30.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:30.193 INFO:teuthology.orchestra.run.smithi098.stdout:5684 write 0xec66659 thru 0xec757ac (0xf154 bytes) 2024-09-16T13:18:30.197 INFO:teuthology.orchestra.run.smithi098.stdout:5686 punch from 0xfee040 to 0xff0434, (0x23f4 bytes) 2024-09-16T13:18:30.197 INFO:teuthology.orchestra.run.smithi098.stdout:5687 write 0xb6fd0b6 thru 0xb705ff1 (0x8f3c bytes) 2024-09-16T13:18:30.203 INFO:teuthology.orchestra.run.smithi098.stdout:5688 write 0x35d6785 thru 0x35da290 (0x3b0c bytes) 2024-09-16T13:18:30.207 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:30.207 INFO:teuthology.orchestra.run.smithi098.stdout:5691 writesame 0x57e05b7 thru 0x57ec469 data_size 0x1(0xbeb3 bytes) 2024-09-16T13:18:30.229 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:30.221+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:30.251 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:30.251 INFO:teuthology.orchestra.run.smithi098.stdout:5694 read 0xce5721b thru 0xce5bbf8 (0x49de bytes) 2024-09-16T13:18:30.254 INFO:teuthology.orchestra.run.smithi098.stdout:5698 read 0x27517f7 thru 0x2753697 (0x1ea1 bytes) 2024-09-16T13:18:30.255 INFO:teuthology.orchestra.run.smithi098.stdout:5700 write 0x87ff0f1 thru 0x8805b83 (0x6a93 bytes) 2024-09-16T13:18:30.262 INFO:teuthology.orchestra.run.smithi098.stdout:5701 read 0x2feda3c thru 0x2fee1c9 (0x78e bytes) 2024-09-16T13:18:30.264 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:30.264 INFO:teuthology.orchestra.run.smithi098.stdout:5703 punch from 0xabf6bca to 0xabf7d8c, (0x11c2 bytes) 2024-09-16T13:18:30.264 INFO:teuthology.orchestra.run.smithi098.stdout:5704 read 0xde4f52f thru 0xde5ad9e (0xb870 bytes) 2024-09-16T13:18:30.264 INFO:teuthology.orchestra.run.smithi098.stdout:5707 write 0x594a6dc thru 0x594e2c1 (0x3be6 bytes) 2024-09-16T13:18:30.270 INFO:teuthology.orchestra.run.smithi098.stdout:5708 read 0x81ad34 thru 0x8236eb (0x89b8 bytes) 2024-09-16T13:18:30.271 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:30.271 INFO:teuthology.orchestra.run.smithi098.stdout:5710 write 0x5776a68 thru 0x57823e8 (0xb981 bytes) 2024-09-16T13:18:30.278 INFO:teuthology.orchestra.run.smithi098.stdout:5711 trunc from 0xec757ad to 0xe2f35f4 2024-09-16T13:18:30.293 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:30.293 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:30.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:30.306 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:18:30.311 INFO:teuthology.orchestra.run.smithi098.stdout:5712 read 0x7e7c6e4 thru 0x7e88120 (0xba3d bytes) 2024-09-16T13:18:30.312 INFO:teuthology.orchestra.run.smithi098.stdout:5714 punch from 0xadcae9f to 0xadd8602, (0xd763 bytes) 2024-09-16T13:18:30.323 INFO:teuthology.orchestra.run.smithi098.stdout:5715 trunc from 0xe2f35f4 to 0x5fe36dd 2024-09-16T13:18:30.329 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:30.323+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:30.363 INFO:teuthology.orchestra.run.smithi098.stdout:5717 write 0x9af5d63 thru 0x9afae16 (0x50b4 bytes) 2024-09-16T13:18:30.370 INFO:teuthology.orchestra.run.smithi098.stdout:5718 read 0x7c0572f thru 0x7c0e763 (0x9035 bytes) 2024-09-16T13:18:30.384 INFO:teuthology.orchestra.run.smithi098.stdout:5719 write 0xaa6270b thru 0xaa6c430 (0x9d26 bytes) 2024-09-16T13:18:30.389 INFO:teuthology.orchestra.run.smithi098.stdout:5720 read 0x81aff5a thru 0x81b903c (0x90e3 bytes) 2024-09-16T13:18:30.389 INFO:teuthology.orchestra.run.smithi098.stdout:5722 punch from 0xa4ccb71 to 0xa4d7884, (0xad13 bytes) 2024-09-16T13:18:30.389 INFO:teuthology.orchestra.run.smithi098.stdout:5724 write 0x61eac78 thru 0x61f5243 (0xa5cc bytes) 2024-09-16T13:18:30.410 INFO:teuthology.orchestra.run.smithi098.stdout:5725 trunc from 0xaa6c431 to 0x32f576f 2024-09-16T13:18:30.411 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:30.411 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:30.422 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:30.439 INFO:teuthology.orchestra.run.smithi098.stdout:5728 punch from 0x181fe83 to 0x1823de2, (0x3f5f bytes) 2024-09-16T13:18:30.439 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:30.433+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:30.462 INFO:teuthology.orchestra.run.smithi098.stdout:5729 write 0xa1b443e thru 0xa1b8419 (0x3fdc bytes) 2024-09-16T13:18:30.467 INFO:teuthology.orchestra.run.smithi098.stdout:5730 write 0x46fa4b8 thru 0x46feda0 (0x48e9 bytes) 2024-09-16T13:18:30.474 INFO:teuthology.orchestra.run.smithi098.stdout:5731 writesame 0x9b8f92b thru 0x9b9f614 data_size 0x42(0xfcea bytes) 2024-09-16T13:18:30.481 INFO:teuthology.orchestra.run.smithi098.stdout:5732 read 0x16f05b8 thru 0x16febe4 (0xe62d bytes) 2024-09-16T13:18:30.498 INFO:teuthology.orchestra.run.smithi098.stdout:5737 write 0xc8434d5 thru 0xc84a220 (0x6d4c bytes) 2024-09-16T13:18:30.505 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:30.518 INFO:teuthology.orchestra.run.smithi098.stdout:5740 write 0xcdac7ff thru 0xcdacedc (0x6de bytes) 2024-09-16T13:18:30.524 INFO:teuthology.orchestra.run.smithi098.stdout:5741 punch from 0x3bfd5a7 to 0x3bff649, (0x20a2 bytes) 2024-09-16T13:18:30.524 INFO:teuthology.orchestra.run.smithi098.stdout:5742 write 0xc8629dd thru 0xc867777 (0x4d9b bytes) 2024-09-16T13:18:30.531 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:30.523+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:30.541 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:30.542 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:30.546 INFO:teuthology.orchestra.run.smithi098.stdout:5743 writesame 0xe7d94d7 thru 0xe7dc27d data_size 0x1d(0x2da7 bytes) 2024-09-16T13:18:30.546 INFO:teuthology.orchestra.run.smithi098.stdout:5745 write 0xd693218 thru 0xd694b46 (0x192f bytes) 2024-09-16T13:18:30.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:30.551 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:30.559 INFO:teuthology.orchestra.run.smithi098.stdout:5747 trunc from 0xe7dc27e to 0xaab461e 2024-09-16T13:18:30.588 INFO:teuthology.orchestra.run.smithi098.stdout:5748 trunc from 0xaab461e to 0xec59790 2024-09-16T13:18:30.599 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:30.599 INFO:teuthology.orchestra.run.smithi098.stdout:5750 write 0xa7db9e3 thru 0xa7e6ceb (0xb309 bytes) 2024-09-16T13:18:30.606 INFO:teuthology.orchestra.run.smithi098.stdout:5751 read 0x72a3f4c thru 0x72add5f (0x9e14 bytes) 2024-09-16T13:18:30.606 INFO:teuthology.orchestra.run.smithi098.stdout:5752 punch from 0x1163318 to 0x1172e60, (0xfb48 bytes) 2024-09-16T13:18:30.606 INFO:teuthology.orchestra.run.smithi098.stdout:5753 read 0x61b2c53 thru 0x61b695f (0x3d0d bytes) 2024-09-16T13:18:30.606 INFO:teuthology.orchestra.run.smithi098.stdout:5754 punch from 0x8145269 to 0x814f395, (0xa12c bytes) 2024-09-16T13:18:30.607 INFO:teuthology.orchestra.run.smithi098.stdout:5756 write 0x92ce899 thru 0x92cfef0 (0x1658 bytes) 2024-09-16T13:18:30.609 INFO:teuthology.orchestra.run.smithi098.stdout:5757 write 0xe1e70e5 thru 0xe1e887a (0x1796 bytes) 2024-09-16T13:18:30.615 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:30.616 INFO:teuthology.orchestra.run.smithi098.stdout:5759 write 0x5c468b9 thru 0x5c49288 (0x29d0 bytes) 2024-09-16T13:18:30.623 INFO:teuthology.orchestra.run.smithi098.stdout:5760 write 0x78c8e0e thru 0x78d6844 (0xda37 bytes) 2024-09-16T13:18:30.631 INFO:teuthology.orchestra.run.smithi098.stdout:5762 read 0x5047132 thru 0x504ac1f (0x3aee bytes) 2024-09-16T13:18:30.631 INFO:teuthology.orchestra.run.smithi098.stdout:5763 writesame 0xadc1a21 thru 0xadc654d data_size 0x5(0x4b2d bytes) 2024-09-16T13:18:30.631 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:30.625+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:30.641 INFO:teuthology.orchestra.run.smithi098.stdout:5764 read 0x47ac510 thru 0x47b855f (0xc050 bytes) 2024-09-16T13:18:30.648 INFO:teuthology.orchestra.run.smithi098.stdout:5765 trunc from 0xec59790 to 0xc24a61c 2024-09-16T13:18:30.664 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:30.664 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:30.665 INFO:teuthology.orchestra.run.smithi098.stdout:5766 read 0x93c2c71 thru 0x93d1d5c (0xf0ec bytes) 2024-09-16T13:18:30.665 INFO:teuthology.orchestra.run.smithi098.stdout:5767 read 0xa8b631d thru 0xa8b64a4 (0x188 bytes) 2024-09-16T13:18:30.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:30.675 INFO:teuthology.orchestra.run.smithi098.stdout:5768 trunc from 0xc24a61c to 0x6a98817 2024-09-16T13:18:30.698 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:30.727 INFO:teuthology.orchestra.run.smithi098.stdout:5772 write 0xe2dd669 thru 0xe2dfbfb (0x2593 bytes) 2024-09-16T13:18:30.732 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:30.725+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:30.734 INFO:teuthology.orchestra.run.smithi098.stdout:5773 write 0x50150be thru 0x50175e8 (0x252b bytes) 2024-09-16T13:18:30.742 INFO:teuthology.orchestra.run.smithi098.stdout:5774 read 0x9cd888e thru 0x9ce715a (0xe8cd bytes) 2024-09-16T13:18:30.742 INFO:teuthology.orchestra.run.smithi098.stdout:5775 read 0xa67115c thru 0xa6737e1 (0x2686 bytes) 2024-09-16T13:18:30.743 INFO:teuthology.orchestra.run.smithi098.stdout:5776 writesame 0xda006dc thru 0xda0b884 data_size 0x1(0xb1a9 bytes) 2024-09-16T13:18:30.769 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:30.770 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:30.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:30.779 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-16T13:18:30.779 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:18:30.780 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-16T13:18:30.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-16T13:18:30.792 INFO:teuthology.orchestra.run.smithi098.stdout:5778 flatten 2024-09-16T13:18:30.809 INFO:teuthology.orchestra.run.smithi098.stdout:5779 write 0xcc3e729 thru 0xcc4123d (0x2b15 bytes) 2024-09-16T13:18:30.812 INFO:teuthology.orchestra.run.smithi098.stdout:5780 write 0x528ec9c thru 0x529460b (0x5970 bytes) 2024-09-16T13:18:30.818 INFO:teuthology.orchestra.run.smithi098.stdout:5781 punch from 0x6a4983f to 0x6a51fde, (0x879f bytes) 2024-09-16T13:18:30.834 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:30.825+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:30.838 INFO:teuthology.orchestra.run.smithi098.stdout:5783 trunc from 0xe2dfbfc to 0x4b7f6d7 2024-09-16T13:18:30.884 INFO:teuthology.orchestra.run.smithi098.stdout:5784 write 0x66cbe8f thru 0x66d422d (0x839f bytes) 2024-09-16T13:18:30.886 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:30.887 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:30.893 INFO:teuthology.orchestra.run.smithi098.stdout:5785 trunc from 0x66d422e to 0x3ae883a 2024-09-16T13:18:30.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:30.915 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:30.915 INFO:teuthology.orchestra.run.smithi098.stdout:5788 read 0x61860 thru 0x62f2b (0x16cc bytes) 2024-09-16T13:18:30.915 INFO:teuthology.orchestra.run.smithi098.stdout:5789 punch from 0x351fdd5 to 0x35230d5, (0x3300 bytes) 2024-09-16T13:18:30.916 INFO:teuthology.orchestra.run.smithi098.stdout:5790 punch from 0x6bb39b to 0x6c62eb, (0xaf50 bytes) 2024-09-16T13:18:30.917 INFO:teuthology.orchestra.run.smithi098.stdout:5791 trunc from 0x3ae883a to 0x2e4e4cd 2024-09-16T13:18:30.934 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:30.927+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:30.935 INFO:teuthology.orchestra.run.smithi098.stdout:5793 read 0x2754fa7 thru 0x2761eeb (0xcf45 bytes) 2024-09-16T13:18:30.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-16T13:18:30.954 INFO:teuthology.orchestra.run.smithi098.stdout:5795 trunc from 0x2e4e4cd to 0xd4a5b74 2024-09-16T13:18:30.965 INFO:teuthology.orchestra.run.smithi098.stdout:5797 clone 39 order 19 su 262144 sc 11 2024-09-16T13:18:31.003 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:31.003 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:31.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:31.036 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:31.029+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:31.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-16T13:18:31.151 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:31.145+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:31.156 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:31.156 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:31.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:31.238 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:31.229+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:31.268 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:31.268 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:31.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:31.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-16T13:18:31.339 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:31.333+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:31.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:31.391 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:31.392 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:31.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:31.440 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:31.433+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:31.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-16T13:18:31.499 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:31.499 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:31.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:31.541 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:31.533+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:31.542 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:31.561 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:31.582 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:31.601 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:31.612 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:31.613 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:31.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:31.625 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:31.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-09-16T13:18:31.642 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:31.635+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:31.652 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #37, image image_client.0-clone37 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent38 2024-09-16T13:18:31.654 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:31.675 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:31.676 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:31.676 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:31.676 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:31.676 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:31.676 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:31.676 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:31.676 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:31.676 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:31.679 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:31.679 INFO:teuthology.orchestra.run.smithi098.stdout:5798 write 0xb60606c thru 0xb612530 (0xc4c5 bytes) 2024-09-16T13:18:31.691 INFO:teuthology.orchestra.run.smithi098.stdout:5799 writesame 0xa476ef2 thru 0xa48096f data_size 0x71(0x9a7e bytes) 2024-09-16T13:18:31.700 INFO:teuthology.orchestra.run.smithi098.stdout:5800 punch from 0x368b1dc to 0x3691b9e, (0x69c2 bytes) 2024-09-16T13:18:31.700 INFO:teuthology.orchestra.run.smithi098.stdout:5801 write 0x5d6b94b thru 0x5d71903 (0x5fb9 bytes) 2024-09-16T13:18:31.706 INFO:teuthology.orchestra.run.smithi098.stdout:5802 write 0x61bf253 thru 0x61ca159 (0xaf07 bytes) 2024-09-16T13:18:31.715 INFO:teuthology.orchestra.run.smithi098.stdout:5804 write 0x827cea8 thru 0x8288861 (0xb9ba bytes) 2024-09-16T13:18:31.727 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:31.727 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:31.728 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:31.738 INFO:teuthology.orchestra.run.smithi098.stdout:5805 trunc from 0xd4a5b74 to 0x426c4e5 2024-09-16T13:18:31.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:31.743 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:31.735+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:31.768 INFO:teuthology.orchestra.run.smithi098.stdout:5806 read 0x37db429 thru 0x37e4013 (0x8beb bytes) 2024-09-16T13:18:31.769 INFO:teuthology.orchestra.run.smithi098.stdout:5807 read 0x36230f3 thru 0x362ed6c (0xbc7a bytes) 2024-09-16T13:18:31.770 INFO:teuthology.orchestra.run.smithi098.stdout:5808 punch from 0x2c50b87 to 0x2c53884, (0x2cfd bytes) 2024-09-16T13:18:31.785 INFO:teuthology.orchestra.run.smithi098.stdout:5809 write 0x705ba47 thru 0x7069fc2 (0xe57c bytes) 2024-09-16T13:18:31.791 INFO:teuthology.orchestra.run.smithi098.stdout:5810 write 0x2ea9291 thru 0x2eafe10 (0x6b80 bytes) 2024-09-16T13:18:31.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-09-16T13:18:31.821 INFO:teuthology.orchestra.run.smithi098.stdout:5811 write 0xbc6a2ea thru 0xbc71b03 (0x781a bytes) 2024-09-16T13:18:31.837 INFO:teuthology.orchestra.run.smithi098.stdout:5812 trunc from 0xbc71b04 to 0x54d371b 2024-09-16T13:18:31.843 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:31.837+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:31.861 INFO:teuthology.orchestra.run.smithi098.stdout:5813 punch from 0x3730a54 to 0x373c2d1, (0xb87d bytes) 2024-09-16T13:18:31.861 INFO:teuthology.orchestra.run.smithi098.stdout:5816 write 0x100ce98 thru 0x101c075 (0xf1de bytes) 2024-09-16T13:18:31.867 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:31.867 INFO:teuthology.orchestra.run.smithi098.stdout:5819 read 0x424bcad thru 0x425344c (0x77a0 bytes) 2024-09-16T13:18:31.868 INFO:teuthology.orchestra.run.smithi098.stdout:5820 writesame 0x47ee866 thru 0x47fa53b data_size 0x15(0xbcd6 bytes) 2024-09-16T13:18:31.875 INFO:teuthology.orchestra.run.smithi098.stdout:5821 read 0x343709e thru 0x343d8c5 (0x6828 bytes) 2024-09-16T13:18:31.880 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:31.880 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:31.880 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:31.880 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:31.880 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:31.886 INFO:teuthology.orchestra.run.smithi098.stdout:5822 trunc from 0x54d371b to 0xab7f496 2024-09-16T13:18:31.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:31.907 INFO:teuthology.orchestra.run.smithi098.stdout:5823 trunc from 0xab7f496 to 0x430bf2a 2024-09-16T13:18:31.939 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:31.942 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2024-09-16T13:18:31.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-16T13:18:31.944 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:31.935+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:31.949 INFO:teuthology.orchestra.run.smithi098.stdout:5824 write 0x89532d7 thru 0x8959309 (0x6033 bytes) 2024-09-16T13:18:31.967 INFO:teuthology.orchestra.run.smithi098.stdout:5825 write 0x9d32f2c thru 0x9d3d7e4 (0xa8b9 bytes) 2024-09-16T13:18:31.973 INFO:teuthology.orchestra.run.smithi098.stdout:5826 trunc from 0x9d3d7e5 to 0x99b7280 2024-09-16T13:18:31.991 INFO:teuthology.orchestra.run.smithi098.stdout:5827 write 0x83951ab thru 0x83a4812 (0xf668 bytes) 2024-09-16T13:18:31.997 INFO:teuthology.orchestra.run.smithi098.stdout:5828 writesame 0x3089c1a thru 0x3096672 data_size 0x1f(0xca59 bytes) 2024-09-16T13:18:32.005 INFO:teuthology.orchestra.run.smithi098.stdout:5829 read 0x1435e7c thru 0x143e69a (0x881f bytes) 2024-09-16T13:18:32.006 INFO:teuthology.orchestra.run.smithi098.stdout:5830 read 0x615fc42 thru 0x61660ef (0x64ae bytes) 2024-09-16T13:18:32.006 INFO:teuthology.orchestra.run.smithi098.stdout:5831 punch from 0x7140af9 to 0x7147330, (0x6837 bytes) 2024-09-16T13:18:32.006 INFO:teuthology.orchestra.run.smithi098.stdout:5832 read 0x409270d thru 0x409a037 (0x792b bytes) 2024-09-16T13:18:32.006 INFO:teuthology.orchestra.run.smithi098.stdout:5833 read 0x8cefe0b thru 0x8cf750a (0x7700 bytes) 2024-09-16T13:18:32.006 INFO:teuthology.orchestra.run.smithi098.stdout:5834 writesame 0x4575a9e thru 0x458265b data_size 0x3(0xcbbe bytes) 2024-09-16T13:18:32.009 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:32.009 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66873 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4034560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.389056758999999, 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034635435999999999, 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.010 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66873 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615785+0000", 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615786+0000", 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615792+0000", 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1900000000000003e-06 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615795+0000", 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9450000000000002e-06 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650232+0000", 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034437669999999997 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650301+0000", 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8329000000000003e-05 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650420+0000", 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119132 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66874 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4042752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.389023553000001, 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034827871000000003, 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66874 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:32.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:32.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615820+0000", 2024-09-16T13:18:32.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.063e-06 2024-09-16T13:18:32.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615826+0000", 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9739999999999999e-06 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615835+0000", 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5920000000000001e-06 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650445+0000", 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034610100999999997 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650512+0000", 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650645+0000", 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013316399999999999 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66875 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4050944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.022 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.389002644000001, 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032338000000003, 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66875 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615839+0000", 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615841+0000", 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9250000000000002e-06 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615846+0000", 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.711e-06 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615851+0000", 2024-09-16T13:18:32.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9579999999999998e-06 2024-09-16T13:18:32.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650676+0000", 2024-09-16T13:18:32.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034824524000000003 2024-09-16T13:18:32.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650758+0000", 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1547999999999999e-05 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650871+0000", 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113244 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.027 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66876 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4059136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.028 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:32.028 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388985749, 2024-09-16T13:18:32.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035239591000000001, 2024-09-16T13:18:32.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.028 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.028 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.028 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.028 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.028 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66876 2024-09-16T13:18:32.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615856+0000", 2024-09-16T13:18:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615858+0000", 2024-09-16T13:18:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.857e-06 2024-09-16T13:18:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615863+0000", 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5389999999999998e-06 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615866+0000", 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650899+0000", 2024-09-16T13:18:32.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032448000000001 2024-09-16T13:18:32.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650973+0000", 2024-09-16T13:18:32.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3627000000000005e-05 2024-09-16T13:18:32.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651095+0000", 2024-09-16T13:18:32.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012239400000000001 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66877 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4067328~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388971048999998, 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035445691000000001, 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.033 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.034 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.034 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.034 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66877 2024-09-16T13:18:32.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.034 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.035 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:32.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:32.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:32.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615872+0000", 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.703e-06 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615877+0000", 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2650000000000004e-06 2024-09-16T13:18:32.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615881+0000", 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.292e-06 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651123+0000", 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035242373 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651198+0000", 2024-09-16T13:18:32.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5096000000000001e-05 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651316+0000", 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011757 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66878 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4075520~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388957156, 2024-09-16T13:18:32.039 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035647261999999999, 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66878 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5400000000000002e-07 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.041 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615890+0000", 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6189999999999999e-06 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615896+0000", 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6529999999999999e-06 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615900+0000", 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0069999999999999e-06 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.042 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651337+0000", 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035437877999999999 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651395+0000", 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7219999999999998e-05 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651531+0000", 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013653100000000001 2024-09-16T13:18:32.043 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66879 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388937814999998, 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035838241999999999, 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.044 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66879 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615904+0000", 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615906+0000", 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615911+0000", 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6679999999999998e-06 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615914+0000", 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.151e-06 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651558+0000", 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035643672000000001 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651624+0000", 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6070999999999998e-05 2024-09-16T13:18:32.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651742+0000", 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011729499999999999 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66880 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4091904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388923235, 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036042148000000003, 2024-09-16T13:18:32.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66880 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2099999999999997e-07 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615924+0000", 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:18:32.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615930+0000", 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6250000000000004e-06 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615933+0000", 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3950000000000001e-06 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.051 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651773+0000", 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035839579000000003 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651846+0000", 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2602999999999996e-05 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651960+0000", 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114319 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66881 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4100096~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388902217999998, 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036242563999999998, 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66881 2024-09-16T13:18:32.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:32.054 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615941+0000", 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.72e-06 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615946+0000", 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.259e-06 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.055 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615950+0000", 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651985+0000", 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036035821000000003 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652058+0000", 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2651999999999995e-05 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.056 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652181+0000", 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123428 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66882 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4108288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388888187999999, 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036437628999999999, 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66882 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:32.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7300000000000002e-07 2024-09-16T13:18:32.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.059 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.059 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615955+0000", 2024-09-16T13:18:32.059 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8199999999999999e-06 2024-09-16T13:18:32.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615961+0000", 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2349999999999996e-06 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615965+0000", 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4620000000000001e-06 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.060 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652207+0000", 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036241873000000001 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652276+0000", 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9105999999999999e-05 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652391+0000", 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011476000000000001 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.061 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66883 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4116480~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388872965000001, 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036642620000000001, 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66883 2024-09-16T13:18:32.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.063 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.77e-07 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615970+0000", 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7519999999999999e-06 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615977+0000", 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9349999999999997e-06 2024-09-16T13:18:32.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615985+0000", 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1389999999999995e-06 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652416+0000", 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036430381999999997 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652487+0000", 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1698000000000004e-05 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652611+0000", 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123337 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66884 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4124672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.066 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388608538, 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036613363000000003, 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66884 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:32.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:32.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.16e-07 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616237+0000", 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5719999999999999e-06 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616245+0000", 2024-09-16T13:18:32.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8820000000000001e-06 2024-09-16T13:18:32.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616251+0000", 2024-09-16T13:18:32.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9630000000000003e-06 2024-09-16T13:18:32.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652638+0000", 2024-09-16T13:18:32.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036387705999999999 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652719+0000", 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0469999999999994e-05 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652846+0000", 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126954 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.072 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66885 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4132864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.072 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:32.072 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388585985999999, 2024-09-16T13:18:32.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036785929000000002, 2024-09-16T13:18:32.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66885 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0999999999999999e-07 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616258+0000", 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0269999999999998e-06 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616263+0000", 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8760000000000002e-06 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616267+0000", 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652873+0000", 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036606376000000003 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652956+0000", 2024-09-16T13:18:32.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2824000000000005e-05 2024-09-16T13:18:32.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653041+0000", 2024-09-16T13:18:32.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4918000000000002e-05 2024-09-16T13:18:32.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.078 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66886 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4141056~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.078 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:32.078 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388569557, 2024-09-16T13:18:32.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078279999999998, 2024-09-16T13:18:32.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.078 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.078 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66886 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616274+0000", 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.73e-06 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616279+0000", 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616283+0000", 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1420000000000002e-06 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653119+0000", 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036836016999999999 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653204+0000", 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5575999999999995e-05 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653350+0000", 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014569499999999999 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.082 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66887 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4149248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388555054000001, 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037308557999999999, 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66887 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.84e-07 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616288+0000", 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7099999999999999e-06 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616294+0000", 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4029999999999998e-06 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616297+0000", 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0079999999999998e-06 2024-09-16T13:18:32.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653375+0000", 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078883 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653455+0000", 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0034e-05 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653595+0000", 2024-09-16T13:18:32.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013913599999999999 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66888 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4157440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388540682999999, 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537323999999997, 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66888 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616301+0000", 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616302+0000", 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616308+0000", 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616317+0000", 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2229999999999996e-06 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653620+0000", 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037302572999999999 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653701+0000", 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1771e-05 2024-09-16T13:18:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653838+0000", 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013640599999999999 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66889 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4165632~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388520154999998, 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037755019000000001, 2024-09-16T13:18:32.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66889 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2300000000000002e-07 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616323+0000", 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.905e-06 2024-09-16T13:18:32.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616329+0000", 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4639999999999997e-06 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616332+0000", 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0180000000000002e-06 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653862+0000", 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037529973000000001 2024-09-16T13:18:32.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653937+0000", 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4954000000000001e-05 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654076+0000", 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013928200000000001 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.095 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66890 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4173824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388504748999999, 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037917073000000003, 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66890 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616337+0000", 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616338+0000", 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616371+0000", 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2806999999999999e-05 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616375+0000", 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.827e-06 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654115+0000", 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.03773986 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654181+0000", 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5652000000000001e-05 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654253+0000", 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2789999999999999e-05 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.099 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.100 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.100 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66891 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4182016~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.100 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:32.100 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388456878, 2024-09-16T13:18:32.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038160044999999997, 2024-09-16T13:18:32.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.100 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66891 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:32.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616385+0000", 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616387+0000", 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.356e-06 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616393+0000", 2024-09-16T13:18:32.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9519999999999999e-06 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616396+0000", 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2959999999999999e-06 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654302+0000", 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037905031999999998 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654395+0000", 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3560000000000005e-05 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654544+0000", 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014916500000000001 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66892 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4190208~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:32.104 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.388441066999999, 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038342319999999999, 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66892 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616407+0000", 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4060000000000002e-06 2024-09-16T13:18:32.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616412+0000", 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-06 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616419+0000", 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.037e-06 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654571+0000", 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038152835000000003 2024-09-16T13:18:32.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654625+0000", 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3643000000000001e-05 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654742+0000", 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117348 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.108 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.109 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:32.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:32.111 INFO:teuthology.orchestra.run.smithi098.stdout:5835 trunc from 0x99b7280 to 0xb02eda3 2024-09-16T13:18:32.111 INFO:teuthology.orchestra.run.smithi098.stdout:5836 trunc from 0xb02eda3 to 0x3079447 2024-09-16T13:18:32.111 INFO:teuthology.orchestra.run.smithi098.stdout:5838 write 0x6e33721 thru 0x6e3d16c (0x9a4c bytes) 2024-09-16T13:18:32.112 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:32.103+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:32.113 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.129 INFO:teuthology.orchestra.run.smithi098.stdout:5840 writesame 0x7f6bf63 thru 0x7f6cc9d data_size 0x1(0xd3b bytes) 2024-09-16T13:18:32.129 INFO:teuthology.orchestra.run.smithi098.stdout:5841 read 0x7d59420 thru 0x7d65b15 (0xc6f6 bytes) 2024-09-16T13:18:32.129 INFO:teuthology.orchestra.run.smithi098.stdout:5842 write 0x7937de1 thru 0x7944cc3 (0xcee3 bytes) 2024-09-16T13:18:32.137 INFO:teuthology.orchestra.run.smithi098.stdout:5843 read 0x3d83803 thru 0x3d86013 (0x2811 bytes) 2024-09-16T13:18:32.137 INFO:teuthology.orchestra.run.smithi098.stdout:5844 read 0x62cba9d thru 0x62d112a (0x568e bytes) 2024-09-16T13:18:32.145 INFO:teuthology.orchestra.run.smithi098.stdout:5845 trunc from 0x7f6cc9e to 0xb5b8b8f 2024-09-16T13:18:32.158 INFO:teuthology.orchestra.run.smithi098.stdout:5846 read 0x9666d37 thru 0x9672e42 (0xc10c bytes) 2024-09-16T13:18:32.158 INFO:teuthology.orchestra.run.smithi098.stdout:5847 write 0x6503083 thru 0x650aed7 (0x7e55 bytes) 2024-09-16T13:18:32.165 INFO:teuthology.orchestra.run.smithi098.stdout:5848 write 0xadbc426 thru 0xadcb296 (0xee71 bytes) 2024-09-16T13:18:32.174 INFO:teuthology.orchestra.run.smithi098.stdout:5849 punch from 0x95f6ad7 to 0x9601bf1, (0xb11a bytes) 2024-09-16T13:18:32.174 INFO:teuthology.orchestra.run.smithi098.stdout:5850 punch from 0x66b5687 to 0x66bec48, (0x95c1 bytes) 2024-09-16T13:18:32.174 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.174 INFO:teuthology.orchestra.run.smithi098.stdout:5853 punch from 0x4d1fc22 to 0x4d27472, (0x7850 bytes) 2024-09-16T13:18:32.174 INFO:teuthology.orchestra.run.smithi098.stdout:5856 writesame 0x67aef94 thru 0x67bc2f9 data_size 0x1(0xd366 bytes) 2024-09-16T13:18:32.211 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:32.203+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:32.226 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:32.227 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:32.227 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:32.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:32.247 INFO:teuthology.orchestra.run.smithi098.stdout:5858 write 0xd903d29 thru 0xd90751f (0x37f7 bytes) 2024-09-16T13:18:32.253 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.253 INFO:teuthology.orchestra.run.smithi098.stdout:5860 writesame 0xd882c23 thru 0xd88bf9b data_size 0x1(0x9379 bytes) 2024-09-16T13:18:32.297 INFO:teuthology.orchestra.run.smithi098.stdout:5861 trunc from 0xd907520 to 0x999dd1a 2024-09-16T13:18:32.311 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:32.303+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:32.318 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:18:32.323 INFO:teuthology.orchestra.run.smithi098.stdout:5862 read 0x1ec10c3 thru 0x1ec3918 (0x2856 bytes) 2024-09-16T13:18:32.340 INFO:teuthology.orchestra.run.smithi098.stdout:5863 write 0xbfb3c7a thru 0xbfbd8b5 (0x9c3c bytes) 2024-09-16T13:18:32.347 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.347 INFO:teuthology.orchestra.run.smithi098.stdout:5865 writesame 0x98a3bba thru 0x98ae692 data_size 0x1(0xaad9 bytes) 2024-09-16T13:18:32.361 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:32.362 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:32.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:32.392 INFO:teuthology.orchestra.run.smithi098.stdout:5868 writesame 0x35e8f85 thru 0x35f71d9 data_size 0x1(0xe255 bytes) 2024-09-16T13:18:32.413 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:32.407+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:32.439 INFO:teuthology.orchestra.run.smithi098.stdout:5869 punch from 0xff78cb to 0x100714e, (0xf883 bytes) 2024-09-16T13:18:32.439 INFO:teuthology.orchestra.run.smithi098.stdout:5870 write 0x622a805 thru 0x623a16b (0xf967 bytes) 2024-09-16T13:18:32.445 INFO:teuthology.orchestra.run.smithi098.stdout:5872 punch from 0x3aadb03 to 0x3ab9253, (0xb750 bytes) 2024-09-16T13:18:32.445 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.445 INFO:teuthology.orchestra.run.smithi098.stdout:5875 read 0x379ab0d thru 0x37aa9c1 (0xfeb5 bytes) 2024-09-16T13:18:32.445 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.445 INFO:teuthology.orchestra.run.smithi098.stdout:5878 punch from 0x975433b to 0x975e6c5, (0xa38a bytes) 2024-09-16T13:18:32.445 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.445 INFO:teuthology.orchestra.run.smithi098.stdout:5883 read 0x3518ecc thru 0x35222cc (0x9401 bytes) 2024-09-16T13:18:32.445 INFO:teuthology.orchestra.run.smithi098.stdout:5884 write 0x790b65d thru 0x7918c9f (0xd643 bytes) 2024-09-16T13:18:32.464 INFO:teuthology.orchestra.run.smithi098.stdout:5885 write 0xd6b35ee thru 0xd6bd5bf (0x9fd2 bytes) 2024-09-16T13:18:32.469 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.469 INFO:teuthology.orchestra.run.smithi098.stdout:5887 read 0xaa32948 thru 0xaa3bf92 (0x964b bytes) 2024-09-16T13:18:32.479 INFO:teuthology.orchestra.run.smithi098.stdout:5888 trunc from 0xd6bd5c0 to 0x80a23b4 2024-09-16T13:18:32.503 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64860 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3899392~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935477578, 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035719773000000003, 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64860 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563336+0000", 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563342+0000", 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3240000000000002e-06 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563345+0000", 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.598973+0000", 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035627631999999999 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599006+0000", 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3495999999999999e-05 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599054+0000", 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7460999999999998e-05 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64861 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3907584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935462696999998, 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035815461, 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64861 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563351+0000", 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.854e-06 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563356+0000", 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5940000000000001e-06 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563360+0000", 2024-09-16T13:18:32.511 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4060000000000001e-06 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599076+0000", 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035716171999999997 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599116+0000", 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9777999999999998e-05 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.512 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599164+0000", 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8307999999999999e-05 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64862 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935448103999999, 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035954097999999997, 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.513 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64862 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:32.514 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563364+0000", 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563365+0000", 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.821e-06 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.515 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563371+0000", 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0769999999999998e-06 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563374+0000", 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3560000000000001e-06 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599187+0000", 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035813153 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.516 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599252+0000", 2024-09-16T13:18:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4873e-05 2024-09-16T13:18:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599317+0000", 2024-09-16T13:18:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5438000000000006e-05 2024-09-16T13:18:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.517 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64863 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3923968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935422549999998, 2024-09-16T13:18:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036051842000000001, 2024-09-16T13:18:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.518 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64863 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.519 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563392+0000", 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3140000000000002e-06 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.520 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563398+0000", 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.066e-06 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563402+0000", 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599339+0000", 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035937678000000001 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599377+0000", 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599441+0000", 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3380999999999995e-05 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.522 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64864 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3932160~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935406145999998, 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036187015000000003, 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64864 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.523 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563406+0000", 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.524 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563411+0000", 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563417+0000", 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563421+0000", 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599458+0000", 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036036870999999998 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599515+0000", 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7067000000000001e-05 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599592+0000", 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7457000000000001e-05 2024-09-16T13:18:32.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64865 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3940352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935386756, 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036265720000000001, 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64865 2024-09-16T13:18:32.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.528 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:18:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563427+0000", 2024-09-16T13:18:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.001e-06 2024-09-16T13:18:32.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563433+0000", 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5489999999999997e-06 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563436+0000", 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.484e-06 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599613+0000", 2024-09-16T13:18:32.530 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036176815000000001 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599646+0000", 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3192999999999999e-05 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599690+0000", 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4240000000000003e-05 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64866 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3948544~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935370978000002, 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036355749999999999, 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64866 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563441+0000", 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:32.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563443+0000", 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0870000000000001e-06 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563448+0000", 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.062e-06 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563451+0000", 2024-09-16T13:18:32.534 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599712+0000", 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036261007999999997 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599753+0000", 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0476999999999999e-05 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.535 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599796+0000", 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3226000000000003e-05 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64867 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3956736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935356238000001, 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036484356000000003, 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.536 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64867 2024-09-16T13:18:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.537 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:32.538 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563456+0000", 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563458+0000", 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9980000000000002e-06 2024-09-16T13:18:32.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563463+0000", 2024-09-16T13:18:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-16T13:18:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.540 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563466+0000", 2024-09-16T13:18:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:18:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599810+0000", 2024-09-16T13:18:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036343683000000002 2024-09-16T13:18:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.541 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599864+0000", 2024-09-16T13:18:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3565e-05 2024-09-16T13:18:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599939+0000", 2024-09-16T13:18:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5889999999999993e-05 2024-09-16T13:18:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64868 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3964928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935341080000001, 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036573319999999999, 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64868 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563471+0000", 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:32.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563473+0000", 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563479+0000", 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.178e-06 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563483+0000", 2024-09-16T13:18:32.545 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6339999999999999e-06 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599963+0000", 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036480090999999999 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600001+0000", 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8340999999999998e-05 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.546 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600044+0000", 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64869 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3973120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935325106000001, 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036707818000000003, 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.547 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64869 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563487+0000", 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9100000000000003e-07 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563489+0000", 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.549 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563494+0000", 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563498+0000", 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.174e-06 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600061+0000", 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036563127000000001 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.550 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600119+0000", 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8476000000000001e-05 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600194+0000", 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4869000000000004e-05 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64870 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3981312~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.551 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935310063999999, 2024-09-16T13:18:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036852528000000002, 2024-09-16T13:18:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64870 2024-09-16T13:18:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563502+0000", 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563504+0000", 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9479999999999998e-06 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563509+0000", 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563512+0000", 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3239999999999999e-06 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600218+0000", 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036705812999999997 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600280+0000", 2024-09-16T13:18:32.555 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2298000000000006e-05 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600354+0000", 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3343000000000006e-05 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64871 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3989504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:32.556 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935295358000001, 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036938311000000001, 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64871 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:32.557 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.558 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563518+0000", 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9130000000000001e-06 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563523+0000", 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563529+0000", 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0710000000000002e-06 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.559 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036847958 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600411+0000", 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4388999999999999e-05 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600454+0000", 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3411000000000002e-05 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.560 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64872 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3997696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935278851, 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037077038999999999, 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64872 2024-09-16T13:18:32.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.562 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563533+0000", 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563535+0000", 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1270000000000001e-06 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563540+0000", 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3120000000000001e-06 2024-09-16T13:18:32.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563544+0000", 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4699999999999998e-06 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600476+0000", 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036931691000000003 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.564 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600543+0000", 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7475e-05 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600610+0000", 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6397999999999997e-05 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64873 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4005888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.565 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935263826, 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037239282999999998, 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64873 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:32.566 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.567 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563550+0000", 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563555+0000", 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563558+0000", 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2009999999999999e-06 2024-09-16T13:18:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600630+0000", 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037071933000000001 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600702+0000", 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1245e-05 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600787+0000", 2024-09-16T13:18:32.569 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5290000000000005e-05 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64874 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4014080~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935249379999998, 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037394846000000002, 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.570 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64874 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0400000000000002e-07 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563565+0000", 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1900000000000002e-06 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563570+0000", 2024-09-16T13:18:32.572 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4199999999999998e-06 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563574+0000", 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600812+0000", 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037237600000000003 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.573 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4690000000000006e-05 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600957+0000", 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0463999999999994e-05 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64875 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4022272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:32.574 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935233584999999, 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537883000000001, 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64875 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.65e-07 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.576 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563580+0000", 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563587+0000", 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2589999999999997e-06 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563591+0000", 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600982+0000", 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037390117 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601044+0000", 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2788999999999998e-05 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601116+0000", 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1282e-05 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64876 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4030464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935216260000001, 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037682571999999998, 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.579 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64876 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563596+0000", 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563613+0000", 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7935999999999999e-05 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563620+0000", 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4740000000000002e-06 2024-09-16T13:18:32.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563624+0000", 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601141+0000", 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037517170000000002 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601203+0000", 2024-09-16T13:18:32.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2187999999999995e-05 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601278+0000", 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.583 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64877 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4038656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935182651000002, 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037813404000000002, 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64877 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563631+0000", 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1330000000000002e-06 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563637+0000", 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563640+0000", 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601304+0000", 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037664438000000001 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601372+0000", 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7618000000000002e-05 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601442+0000", 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-05 2024-09-16T13:18:32.587 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64878 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4046848~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935153478, 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037956456, 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64878 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563659+0000", 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563661+0000", 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-06 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563667+0000", 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3470000000000004e-06 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563671+0000", 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7170000000000002e-06 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601473+0000", 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037801955999999998 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601537+0000", 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3960000000000004e-05 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.591 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601614+0000", 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64879 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4055040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.935136838999998, 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038485017000000003, 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.592 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64879 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.593 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4099999999999998e-07 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563677+0000", 2024-09-16T13:18:32.594 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1299999999999999e-06 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563682+0000", 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563688+0000", 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3329999999999996e-06 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601640+0000", 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037951864000000002 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601705+0000", 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5573000000000004e-05 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.602160+0000", 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000454411 2024-09-16T13:18:32.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.597 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:5890 write 0x40f58f3 thru 0x41053aa (0xfab8 bytes) 2024-09-16T13:18:32.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:5891 write 0x42bffdc thru 0x42cf16f (0xf194 bytes) 2024-09-16T13:18:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:5894 read 0x1753b86 thru 0x1753e40 (0x2bb bytes) 2024-09-16T13:18:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:5895 writesame 0x4c5cc37 thru 0x4c5d731 data_size 0x3(0xafb bytes) 2024-09-16T13:18:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:5896 read 0x78cac09 thru 0x78d36e9 (0x8ae1 bytes) 2024-09-16T13:18:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:5897 write 0x6db3d38 thru 0x6dbcc71 (0x8f3a bytes) 2024-09-16T13:18:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:5899 writesame 0xdfeb28c thru 0xdff986b data_size 0x30(0xe5e0 bytes) 2024-09-16T13:18:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:5901 read 0x7bc512f thru 0x7bca6bc (0x558e bytes) 2024-09-16T13:18:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:5904 read 0x1bce894 thru 0x1bd6842 (0x7faf bytes) 2024-09-16T13:18:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:5905 write 0x94c660e thru 0x94c78ee (0x12e1 bytes) 2024-09-16T13:18:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:5906 write 0x8fd17ab thru 0x8fd8ed1 (0x7727 bytes) 2024-09-16T13:18:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:5907 punch from 0xd4d2e7 to 0xd4e98c, (0x16a5 bytes) 2024-09-16T13:18:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:5908 trunc from 0xdff986c to 0x76e63aa 2024-09-16T13:18:32.602 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:32.591+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:32.605 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:32.620 INFO:teuthology.orchestra.run.smithi098.stdout:5910 writesame 0x78d971a thru 0x78e78f3 data_size 0x1(0xe1da bytes) 2024-09-16T13:18:32.644 INFO:teuthology.orchestra.run.smithi098.stdout:5911 writesame 0xe9b7416 thru 0xe9bbb5e data_size 0xb(0x4749 bytes) 2024-09-16T13:18:32.644 INFO:teuthology.orchestra.run.smithi098.stdout:5913 trunc from 0xe9bbb5f to 0xea22034 2024-09-16T13:18:32.655 INFO:teuthology.orchestra.run.smithi098.stdout:5914 write 0xb74f40c thru 0xb7599a4 (0xa599 bytes) 2024-09-16T13:18:32.683 INFO:teuthology.orchestra.run.smithi098.stdout:5915 trunc from 0xea22034 to 0x161ed01 2024-09-16T13:18:32.699 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:32.693+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:32.705 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:32.706 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:32.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:32.722 INFO:teuthology.orchestra.run.smithi098.stdout:5917 trunc from 0x161ed01 to 0x2f6614c 2024-09-16T13:18:32.748 INFO:teuthology.orchestra.run.smithi098.stdout:5918 write 0x3c8ceeb thru 0x3c99a6f (0xcb85 bytes) 2024-09-16T13:18:32.786 INFO:teuthology.orchestra.run.smithi098.stdout:5920 write 0xca4ae16 thru 0xca50a99 (0x5c84 bytes) 2024-09-16T13:18:32.792 INFO:teuthology.orchestra.run.smithi098.stdout:5923 write 0x49f1776 thru 0x49f3970 (0x21fb bytes) 2024-09-16T13:18:32.798 INFO:teuthology.orchestra.run.smithi098.stdout:5924 writesame 0x7bdc844 thru 0x7be199a data_size 0xb(0x5157 bytes) 2024-09-16T13:18:32.800 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:32.793+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:32.807 INFO:teuthology.orchestra.run.smithi098.stdout:5925 trunc from 0xca50a9a to 0xd3a04d4 2024-09-16T13:18:32.811 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:32.811 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:32.812 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:32.820 INFO:teuthology.orchestra.run.smithi098.stdout:5927 read 0x9f85186 thru 0x9f8e516 (0x9391 bytes) 2024-09-16T13:18:32.820 INFO:teuthology.orchestra.run.smithi098.stdout:5929 punch from 0xa310648 to 0xa31a477, (0x9e2f bytes) 2024-09-16T13:18:32.820 INFO:teuthology.orchestra.run.smithi098.stdout:5931 write 0x2cadb86 thru 0x2caf72e (0x1ba9 bytes) 2024-09-16T13:18:32.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:32.826 INFO:teuthology.orchestra.run.smithi098.stdout:5932 punch from 0xda3ace to 0xdaa9ef, (0x6f21 bytes) 2024-09-16T13:18:32.827 INFO:teuthology.orchestra.run.smithi098.stdout:5934 read 0xa10b0e0 thru 0xa114b7b (0x9a9c bytes) 2024-09-16T13:18:32.827 INFO:teuthology.orchestra.run.smithi098.stdout:5936 read 0x4970044 thru 0x4971c29 (0x1be6 bytes) 2024-09-16T13:18:32.840 INFO:teuthology.orchestra.run.smithi098.stdout:5937 write 0xe04dfeb thru 0xe05d5f1 (0xf607 bytes) 2024-09-16T13:18:32.846 INFO:teuthology.orchestra.run.smithi098.stdout:5938 write 0x982d0e0 thru 0x983caa7 (0xf9c8 bytes) 2024-09-16T13:18:32.853 INFO:teuthology.orchestra.run.smithi098.stdout:5939 write 0x8e6d5e3 thru 0x8e76c65 (0x9683 bytes) 2024-09-16T13:18:32.862 INFO:teuthology.orchestra.run.smithi098.stdout:5941 read 0x192e380 thru 0x19367ac (0x842d bytes) 2024-09-16T13:18:32.862 INFO:teuthology.orchestra.run.smithi098.stdout:5942 read 0xb179ac4 thru 0xb17dda2 (0x42df bytes) 2024-09-16T13:18:32.862 INFO:teuthology.orchestra.run.smithi098.stdout:5943 writesame 0x5076bc6 thru 0x5083e6b data_size 0x3b(0xd2a6 bytes) 2024-09-16T13:18:32.872 INFO:teuthology.orchestra.run.smithi098.stdout:5945 writesame 0xa05a5fd thru 0xa068226 data_size 0x1(0xdc2a bytes) 2024-09-16T13:18:32.901 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:32.893+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:32.929 INFO:teuthology.orchestra.run.smithi098.stdout:5946 write 0x384d6f8 thru 0x385478e (0x7097 bytes) 2024-09-16T13:18:32.930 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 88.916774986999997, 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:18:32.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:18:32.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:18:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:18:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:18:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:18:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:18:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.936 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 88.915989487999994, 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:18:32.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:18:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:18:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:18:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:18:32.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:18:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:18:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:18:32.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:18:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 88.914822565999998, 2024-09-16T13:18:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:18:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.946 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2452 3.9 e31/23 3:9423f6c3:::rbd_data.112dbfd5d823.0000000000000203:head v 31'20, mlcod=31'17)", 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 88.908934195, 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0033299409999999999, 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015233+0000", 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015234+0000", 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6099999999999999e-07 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015256+0000", 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1848000000000002e-05 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015274+0000", 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8328000000000001e-05 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015282+0000", 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0790000000000001e-06 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015338+0000", 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5223999999999998e-05 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.015365+0000", 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7849999999999999e-05 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018527+0000", 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003161348 2024-09-16T13:18:32.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.018563+0000", 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6402999999999997e-05 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2918 3.4 e31/23 3:221e06ec:::rbd_data.112dbfd5d823.000000000000026a:head v 31'12, mlcod=31'10)", 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 88.117581103000006, 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034098290000000001, 2024-09-16T13:18:32.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806586+0000", 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806588+0000", 2024-09-16T13:18:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.12e-06 2024-09-16T13:18:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806604+0000", 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5565000000000001e-05 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806641+0000", 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6536999999999999e-05 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.806657+0000", 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6328e-05 2024-09-16T13:18:32.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807036+0000", 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000379262 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.807057+0000", 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0463e-05 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809955+0000", 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028983759999999998 2024-09-16T13:18:32.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.809996+0000", 2024-09-16T13:18:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1177999999999998e-05 2024-09-16T13:18:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:18:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:32.957 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.865478800000005, 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:18:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:18:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:18:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:18:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:18:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:18:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:18:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:18:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:18:32.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.86486936, 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.963 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:18:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:18:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:18:32.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:18:32.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:32.968 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.367585546000001, 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:18:32.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.973 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.366484729999996, 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:18:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:18:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:18:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:18:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:18:32.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.366345089999996, 2024-09-16T13:18:32.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:32.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:18:32.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:18:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:18:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.983 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.365844559999999, 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:18:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:18:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:18:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:18:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:18:32.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:18:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 75.370675173999999, 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:32.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:18:32.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.993 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.114261333000002, 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:18:32.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 64.501698226000002, 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:18:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:18:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 37.663592055999999, 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:18:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 37.299464723, 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:18:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:75420 3.4 e88/23 3:2605e3b8:::rbd_object_map.112d9326a5dd.0000000000000026:head v 88'165, mlcod=88'163)", 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.420735707, 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055875079999999997, 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:33.011 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503433+0000", 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503451+0000", 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7640999999999999e-05 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503476+0000", 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5599999999999999e-05 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503495+0000", 2024-09-16T13:18:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.825e-05 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503611+0000", 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011631799999999999 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503644+0000", 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3476000000000003e-05 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.508954+0000", 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053093799999999998 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.509019+0000", 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5425999999999993e-05 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:80574 3.4 e90/23 3:24bef2c8:::rbd_data.112da28a57db.0000000000000020:head v 90'178, mlcod=89'174)", 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:22.376510+0000", 2024-09-16T13:18:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.547657124000001, 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034412169999999999, 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376510+0000", 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376510+0000", 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376512+0000", 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.463e-06 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376524+0000", 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2249000000000001e-05 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376578+0000", 2024-09-16T13:18:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3672999999999999e-05 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376591+0000", 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3358e-05 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376659+0000", 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7492999999999999e-05 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.376690+0000", 2024-09-16T13:18:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1378999999999999e-05 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.379911+0000", 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003220728 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:22.379952+0000", 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0874000000000002e-05 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:82268 3.4 e90/23 3:27686806:::rbd_data.112d12abb20f.000000000000001b:head v 90'182, mlcod=90'180)", 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:23.720888+0000", 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.2032795919999995, 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003148812, 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720888+0000", 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720888+0000", 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720889+0000", 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.344e-06 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720900+0000", 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0942e-05 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720924+0000", 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4128999999999999e-05 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.720935+0000", 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1141000000000001e-05 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.721028+0000", 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2785e-05 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.721063+0000", 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4375000000000002e-05 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.723978+0000", 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0029157710000000002 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:23.724037+0000", 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8325000000000001e-05 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:82956 3.0 e90/23 3:02b9fb4a:::rbd_data.112d12abb20f.0000000000000085:head v 90'317, mlcod=90'315)", 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 8.7893247080000005, 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035271679999999998, 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4700000000000001e-07 2024-09-16T13:18:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134851+0000", 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7770000000000002e-06 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134863+0000", 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1705000000000001e-05 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134869+0000", 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6829999999999996e-06 2024-09-16T13:18:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134994+0000", 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012497300000000001 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.135038+0000", 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4548000000000001e-05 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.138295+0000", 2024-09-16T13:18:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032565070000000001 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.138370+0000", 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5228000000000005e-05 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.028 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:33.029 INFO:teuthology.orchestra.run.smithi098.stdout:5947 punch from 0x5966196 to 0x596715f, (0xfc9 bytes) 2024-09-16T13:18:33.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:33.030 INFO:teuthology.orchestra.run.smithi098.stdout:5948 trunc from 0xe05d5f2 to 0x401ddf8 2024-09-16T13:18:33.031 INFO:teuthology.orchestra.run.smithi098.stdout:5949 read 0x34853d7 thru 0x3492ea1 (0xdacb bytes) 2024-09-16T13:18:33.031 INFO:teuthology.orchestra.run.smithi098.stdout:5951 write 0xa8ce225 thru 0xa8d55db (0x73b7 bytes) 2024-09-16T13:18:33.031 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:33.023+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:33.036 INFO:teuthology.orchestra.run.smithi098.stdout:5952 writesame 0xd566709 thru 0xd568a27 data_size 0x25(0x231f bytes) 2024-09-16T13:18:33.036 INFO:teuthology.orchestra.run.smithi098.stdout:5953 write 0x5c10cf0 thru 0x5c205d6 (0xf8e7 bytes) 2024-09-16T13:18:33.068 INFO:teuthology.orchestra.run.smithi098.stdout:5954 trunc from 0xd568a28 to 0x24366ba 2024-09-16T13:18:33.099 INFO:teuthology.orchestra.run.smithi098.stdout:5955 trunc from 0x24366ba to 0x69c6acf 2024-09-16T13:18:33.109 INFO:teuthology.orchestra.run.smithi098.stdout:5957 read 0x46d0e86 thru 0x46d48a0 (0x3a1b bytes) 2024-09-16T13:18:33.130 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:33.121+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:33.136 INFO:teuthology.orchestra.run.smithi098.stdout:5958 writesame 0xe18ffc6 thru 0xe199f63 data_size 0x1(0x9f9e bytes) 2024-09-16T13:18:33.136 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:33.136 INFO:teuthology.orchestra.run.smithi098.stdout:5960 read 0x61c6d68 thru 0x61c72f8 (0x591 bytes) 2024-09-16T13:18:33.136 INFO:teuthology.orchestra.run.smithi098.stdout:5961 writesame 0x204dcf1 thru 0x20579f2 data_size 0xae(0x9d02 bytes) 2024-09-16T13:18:33.143 INFO:teuthology.orchestra.run.smithi098.stdout:5962 write 0xc6dff5 thru 0xc6ed09 (0xd15 bytes) 2024-09-16T13:18:33.147 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:33.147 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:33.147 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:33.153 INFO:teuthology.orchestra.run.smithi098.stdout:5964 write 0x39ed336 thru 0x39f17f7 (0x44c2 bytes) 2024-09-16T13:18:33.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:33.159 INFO:teuthology.orchestra.run.smithi098.stdout:5965 read 0xa54d8f thru 0xa615c4 (0xc836 bytes) 2024-09-16T13:18:33.161 INFO:teuthology.orchestra.run.smithi098.stdout:5967 write 0x71986fb thru 0x719e9ae (0x62b4 bytes) 2024-09-16T13:18:33.168 INFO:teuthology.orchestra.run.smithi098.stdout:5968 write 0x67cb8bc thru 0x67d3f62 (0x86a7 bytes) 2024-09-16T13:18:33.173 INFO:teuthology.orchestra.run.smithi098.stdout:5969 write 0x2f530ef thru 0x2f61af3 (0xea05 bytes) 2024-09-16T13:18:33.180 INFO:teuthology.orchestra.run.smithi098.stdout:5970 read 0xabf8557 thru 0xac0759b (0xf045 bytes) 2024-09-16T13:18:33.180 INFO:teuthology.orchestra.run.smithi098.stdout:5971 punch from 0x280ae0a to 0x280b447, (0x63d bytes) 2024-09-16T13:18:33.180 INFO:teuthology.orchestra.run.smithi098.stdout:5972 flatten 2024-09-16T13:18:33.226 INFO:teuthology.orchestra.run.smithi098.stdout:5973 write 0xb36c8a thru 0xb3c83f (0x5bb6 bytes) 2024-09-16T13:18:33.228 INFO:teuthology.orchestra.run.smithi098.stdout:5974 read 0x2b21211 thru 0x2b28d33 (0x7b23 bytes) 2024-09-16T13:18:33.228 INFO:teuthology.orchestra.run.smithi098.stdout:5975 write 0x3c28478 thru 0x3c2969a (0x1223 bytes) 2024-09-16T13:18:33.230 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:33.225+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:33.233 INFO:teuthology.orchestra.run.smithi098.stdout:5976 write 0xd84be53 thru 0xd850249 (0x43f7 bytes) 2024-09-16T13:18:33.238 INFO:teuthology.orchestra.run.smithi098.stdout:5977 write 0x6ef3b11 thru 0x6efecbe (0xb1ae bytes) 2024-09-16T13:18:33.242 INFO:teuthology.orchestra.run.smithi098.stdout:5978 punch from 0x974ac36 to 0x9756e74, (0xc23e bytes) 2024-09-16T13:18:33.242 INFO:teuthology.orchestra.run.smithi098.stdout:5979 read 0xb83b9b thru 0xb92c82 (0xf0e8 bytes) 2024-09-16T13:18:33.242 INFO:teuthology.orchestra.run.smithi098.stdout:5981 punch from 0x659f0aa to 0x65a6ad7, (0x7a2d bytes) 2024-09-16T13:18:33.242 INFO:teuthology.orchestra.run.smithi098.stdout:5982 read 0xb9d15f0 thru 0xb9e137b (0xfd8c bytes) 2024-09-16T13:18:33.242 INFO:teuthology.orchestra.run.smithi098.stdout:5983 write 0x9400f85 thru 0x940a55c (0x95d8 bytes) 2024-09-16T13:18:33.252 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:33.252 INFO:teuthology.orchestra.run.smithi098.stdout:5987 read 0xd08e0df thru 0xd08f8d5 (0x17f7 bytes) 2024-09-16T13:18:33.252 INFO:teuthology.orchestra.run.smithi098.stdout:5988 read 0x96aa5e thru 0x975f37 (0xb4da bytes) 2024-09-16T13:18:33.261 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:33.261 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:33.269 INFO:teuthology.orchestra.run.smithi098.stdout:5989 trunc from 0xe199f64 to 0x494cae8 2024-09-16T13:18:33.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:33.291 INFO:teuthology.orchestra.run.smithi098.stdout:5990 punch from 0x1b8f777 to 0x1b9067f, (0xf08 bytes) 2024-09-16T13:18:33.311 INFO:teuthology.orchestra.run.smithi098.stdout:5994 write 0x9ffa339 thru 0x9fff465 (0x512d bytes) 2024-09-16T13:18:33.322 INFO:teuthology.orchestra.run.smithi098.stdout:5996 read 0x8a53e3c thru 0x8a5f31a (0xb4df bytes) 2024-09-16T13:18:33.322 INFO:teuthology.orchestra.run.smithi098.stdout:5999 read 0x457e6a0 thru 0x458d168 (0xeac9 bytes) 2024-09-16T13:18:33.322 INFO:teuthology.orchestra.run.smithi098.stdout:compare and write disabled 2024-09-16T13:18:33.331 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:33.325+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:33.335 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:18:33.356 INFO:teuthology.orchestra.run.smithi098.stdout:checking clone #38, image image_client.0-clone38 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent39 2024-09-16T13:18:33.401 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:33.401 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62278 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080190314999999, 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027907498999999999, 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62278 2024-09-16T13:18:33.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315829+0000", 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.79e-06 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315835+0000", 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.682e-06 2024-09-16T13:18:33.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315838+0000", 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0730000000000001e-06 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343640+0000", 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027802384999999999 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343683+0000", 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3058e-05 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343734+0000", 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1104999999999999e-05 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080175753999999, 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:18:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080151787999998, 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:18:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:18:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:18:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080136618000001, 2024-09-16T13:18:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:18:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:18:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:18:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:18:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:18:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:18:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:18:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:18:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:18:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:18:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080122195000001, 2024-09-16T13:18:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:18:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:18:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:18:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:18:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:18:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:18:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:18:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:18:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:18:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:18:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080100949999999, 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:18:33.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080086381000001, 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:18:33.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:18:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:18:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:18:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:18:33.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:18:33.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080067612000001, 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080052717000001, 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:18:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:18:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:18:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:18:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:18:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:18:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080038506000001, 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:18:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:18:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:18:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:18:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:18:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:18:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:18:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:18:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:18:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080025137, 2024-09-16T13:18:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:18:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:18:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:18:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:18:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:18:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:18:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:18:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:18:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:18:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:18:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:18:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:18:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:18:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.080010455, 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:18:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.464 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.079995285999999, 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.468 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.079980645999999, 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:18:33.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.473 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.079949918000001, 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:18:33.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.079928900999999, 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.478 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:18:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:18:33.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:18:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:18:33.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:18:33.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.483 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.079908793000001, 2024-09-16T13:18:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:18:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.484 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:18:33.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:18:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:18:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.489 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.079888889999999, 2024-09-16T13:18:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:18:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:33.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:18:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:18:33.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:18:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:18:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.495 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.079873676999998, 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.496 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 20.079858142999999, 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:33.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:33.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:18:33.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:18:33.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:18:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:33.505 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:33.505 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:33.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:33.507 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:33.499+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:33.608 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:33.599+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:33.615 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:33.615 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:33.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:33.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:33.708 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:33.701+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:33.724 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:33.724 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:33.724 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:33.724 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:33.725 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:33.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:33.809 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:33.801+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:33.816 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:33.837 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:33.838 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:33.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:33.838 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:33.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.838 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:33.838 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:33.838 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 20.622952839, 2024-09-16T13:18:33.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:33.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:33.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:18:33.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:18:33.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:18:33.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:18:33.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:18:33.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:18:33.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:18:33.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:18:33.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:18:33.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:18:33.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:33.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:18:33.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:18:33.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75320 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1114112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.377713113, 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048391086999999999, 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.845 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75320 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455024+0000", 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455026+0000", 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455033+0000", 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8630000000000002e-06 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455037+0000", 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.608e-06 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503178+0000", 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048141646000000003 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503284+0000", 2024-09-16T13:18:33.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105917 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503415+0000", 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130359 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75321 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1441792~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:33.849 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.377696121, 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048644984000000002, 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75321 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455041+0000", 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455043+0000", 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0480000000000001e-06 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455048+0000", 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455052+0000", 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000002e-06 2024-09-16T13:18:33.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503461+0000", 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048409079000000001 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503580+0000", 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119234 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503685+0000", 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105006 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75322 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 2064384~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.377670724, 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048835044000000001, 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.854 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75322 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2599999999999998e-07 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455068+0000", 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.092e-06 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455074+0000", 2024-09-16T13:18:33.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.925e-06 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455078+0000", 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503740+0000", 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048661514000000003 2024-09-16T13:18:33.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503822+0000", 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2196999999999999e-05 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503901+0000", 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9091000000000006e-05 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75344 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 589824~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.377167084, 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048412121000000002, 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75344 2024-09-16T13:18:33.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455570+0000", 2024-09-16T13:18:33.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:18:33.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455572+0000", 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2749999999999998e-06 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455578+0000", 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455582+0000", 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4789999999999998e-06 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503719+0000", 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048137054999999998 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503819+0000", 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9938000000000004e-05 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503982+0000", 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016225500000000001 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75345 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 917504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.377150417999999, 2024-09-16T13:18:33.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048702503000000001, 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75345 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455587+0000", 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455589+0000", 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455594+0000", 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6840000000000001e-06 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455598+0000", 2024-09-16T13:18:33.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7009999999999998e-06 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504027+0000", 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048429382999999999 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504123+0000", 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5251e-05 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504289+0000", 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016599200000000001 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75346 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1212416~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.377135061000001, 2024-09-16T13:18:33.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048896186000000001, 2024-09-16T13:18:33.873 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.873 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:33.873 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.873 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.873 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.873 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75346 2024-09-16T13:18:33.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.873 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455602+0000", 2024-09-16T13:18:33.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2799999999999996e-07 2024-09-16T13:18:33.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455604+0000", 2024-09-16T13:18:33.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:33.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455609+0000", 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455615+0000", 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2020000000000003e-06 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504315+0000", 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048700565000000001 2024-09-16T13:18:33.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504382+0000", 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7088999999999994e-05 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504498+0000", 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115408 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.877 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75347 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1540096~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.377116515000001, 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049121102999999999, 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.878 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75347 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7100000000000002e-07 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455623+0000", 2024-09-16T13:18:33.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:18:33.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455629+0000", 2024-09-16T13:18:33.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:18:33.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455633+0000", 2024-09-16T13:18:33.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9400000000000004e-06 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504546+0000", 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048912718000000001 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504623+0000", 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7726999999999996e-05 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504741+0000", 2024-09-16T13:18:33.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117874 2024-09-16T13:18:33.883 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.883 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.883 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.883 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75353 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 98304~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.883 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:33.884 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376980770999999, 2024-09-16T13:18:33.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048929180000000003, 2024-09-16T13:18:33.884 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.884 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:33.884 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.884 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.884 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.884 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75353 2024-09-16T13:18:33.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.885 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:33.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455758+0000", 2024-09-16T13:18:33.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8789999999999999e-06 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455763+0000", 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2190000000000003e-06 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455767+0000", 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504529+0000", 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048761566999999999 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504587+0000", 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8564999999999999e-05 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504685+0000", 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7754e-05 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75354 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 393216~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376945701, 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049086520000000002, 2024-09-16T13:18:33.889 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75354 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455794+0000", 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.508e-06 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455801+0000", 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5630000000000003e-06 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455805+0000", 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504704+0000", 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048898981000000001 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504759+0000", 2024-09-16T13:18:33.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4330000000000003e-05 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504877+0000", 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118608 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75355 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:33.894 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376927069000001, 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118086999999998, 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75355 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455810+0000", 2024-09-16T13:18:33.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455812+0000", 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455818+0000", 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455821+0000", 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504914+0000", 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049092961999999997 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504928+0000", 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3210000000000001e-05 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75356 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376911308, 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049129843999999999, 2024-09-16T13:18:33.899 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75356 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:33.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455826+0000", 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0399999999999996e-07 2024-09-16T13:18:33.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455828+0000", 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-06 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455835+0000", 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8410000000000002e-06 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455839+0000", 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2400000000000001e-06 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504947+0000", 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049108546000000003 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504955+0000", 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6109999999999996e-06 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75357 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376894044, 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049141246, 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75357 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455843+0000", 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999998e-07 2024-09-16T13:18:33.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455845+0000", 2024-09-16T13:18:33.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999998e-06 2024-09-16T13:18:33.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455852+0000", 2024-09-16T13:18:33.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:18:33.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455856+0000", 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6059999999999999e-06 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504974+0000", 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118385000000001 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504984+0000", 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7880000000000001e-06 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75358 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376877094999999, 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049149627000000001, 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:33.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.909 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.909 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.909 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75358 2024-09-16T13:18:33.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.909 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:33.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455860+0000", 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455862+0000", 2024-09-16T13:18:33.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:18:33.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455867+0000", 2024-09-16T13:18:33.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.395e-06 2024-09-16T13:18:33.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455873+0000", 2024-09-16T13:18:33.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0120000000000002e-06 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505002+0000", 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049128988999999998 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505009+0000", 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7950000000000002e-06 2024-09-16T13:18:33.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75370 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376859341999999, 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049174830000000003, 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:33.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.914 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.914 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.914 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75370 2024-09-16T13:18:33.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.914 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:33.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455878+0000", 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455880+0000", 2024-09-16T13:18:33.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:33.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455885+0000", 2024-09-16T13:18:33.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:33.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455889+0000", 2024-09-16T13:18:33.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6090000000000001e-06 2024-09-16T13:18:33.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505040+0000", 2024-09-16T13:18:33.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049151242999999997 2024-09-16T13:18:33.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505052+0000", 2024-09-16T13:18:33.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2055999999999999e-05 2024-09-16T13:18:33.918 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.918 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.918 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75371 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376844205999999, 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049181864999999998, 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75371 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455893+0000", 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455895+0000", 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455900+0000", 2024-09-16T13:18:33.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.536e-06 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455904+0000", 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505067+0000", 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049163224999999998 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505074+0000", 2024-09-16T13:18:33.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75372 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376829067999999, 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049214870000000001, 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.923 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75372 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:33.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455908+0000", 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.89e-07 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455910+0000", 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455915+0000", 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455919+0000", 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.472e-06 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505109+0000", 2024-09-16T13:18:33.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049190747 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505122+0000", 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3009e-05 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75373 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.927 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376814551000001, 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049224933999999998, 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75373 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455924+0000", 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455930+0000", 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4929999999999998e-06 2024-09-16T13:18:33.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455936+0000", 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4459999999999998e-06 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505140+0000", 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049203510999999998 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505147+0000", 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1670000000000002e-06 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75374 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376796472000001, 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049235427999999998, 2024-09-16T13:18:33.932 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75374 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:33.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455946+0000", 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860000000000001e-06 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455953+0000", 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5119999999999997e-06 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455956+0000", 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.579e-06 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505166+0000", 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049209527000000003 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505175+0000", 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.936 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75375 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.376776191999999, 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049243375999999998, 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75375 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455961+0000", 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455963+0000", 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729999999999999e-06 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455970+0000", 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2520000000000002e-06 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455975+0000", 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505198+0000", 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049222852999999997 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505204+0000", 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:18:33.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.941 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:33.941 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:33.941 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:33.943 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:33.943 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:33.943 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:33.943 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:33.943 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:33.943 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:33.933+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:33.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:33.944 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:33.948 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:34.043 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:34.037+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:34.046 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:34.046 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:34.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:34.145 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:34.137+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:34.159 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:34.159 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:34.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:34.246 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:34.237+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:34.262 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:34.262 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:34.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:34.346 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:34.341+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:34.378 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:34.379 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:34.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:34.447 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:34.441+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:34.485 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:34.486 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:34.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:34.549 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:34.541+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:34.594 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:34.594 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:34.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:34.655 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:34.649+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:34.712 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:34.712 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:34.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:34.751 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:34.745+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:34.819 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:34.820 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:34.828 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:34.871 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:34.865+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:34.952 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:34.947+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:34.976 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:34.976 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:34.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:35.053 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:35.045+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:35.087 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:35.088 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:35.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:35.153 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:35.147+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:35.243 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:35.243 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:35.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:35.254 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:35.245+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:35.346 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:35.347 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:35.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:18:35.354 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:35.347+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:35.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:35.455 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:35.455 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:35.455 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:35.447+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:35.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:35.556 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:35.551+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:35.577 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:35.577 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:35.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:35.657 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:35.649+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:35.679 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:35.679 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:35.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:35.758 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:35.753+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:35.786 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:35.786 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:35.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:35.859 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:35.853+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:35.890 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:35.891 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:35.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:35.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:35.960 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:35.953+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:36.045 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:36.045 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:36.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:36.076 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:36.077 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:36.069+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:36.097 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:36.117 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:36.135 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:36.156 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:36.162 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:36.155+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:36.163 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:36.163 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:36.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:36.182 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:36.203 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:36.205 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:36.209 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:36.263 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:36.255+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:36.276 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:36.276 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:36.276 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:36.276 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:36.276 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:36.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:36.365 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:36.357+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:36.370 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:18:36.397 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:36.397 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:36.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:36.397 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:36.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.397 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:36.397 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:36.397 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.183790115000001, 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:18:36.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:18:36.402 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75320 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1114112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.938550389, 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048391086999999999, 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75320 2024-09-16T13:18:36.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455024+0000", 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455026+0000", 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455033+0000", 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8630000000000002e-06 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455037+0000", 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.608e-06 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503178+0000", 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048141646000000003 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503284+0000", 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105917 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503415+0000", 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130359 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75321 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1441792~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.938533397, 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048644984000000002, 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.407 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.408 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.408 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.408 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75321 2024-09-16T13:18:36.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.408 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455041+0000", 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455043+0000", 2024-09-16T13:18:36.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0480000000000001e-06 2024-09-16T13:18:36.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455048+0000", 2024-09-16T13:18:36.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-16T13:18:36.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455052+0000", 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000002e-06 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503461+0000", 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048409079000000001 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503580+0000", 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119234 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503685+0000", 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105006 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75322 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 2064384~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.938507999999999, 2024-09-16T13:18:36.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048835044000000001, 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75322 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2599999999999998e-07 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455068+0000", 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.092e-06 2024-09-16T13:18:36.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455074+0000", 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.925e-06 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455078+0000", 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503740+0000", 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048661514000000003 2024-09-16T13:18:36.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503822+0000", 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2196999999999999e-05 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503901+0000", 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9091000000000006e-05 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75344 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 589824~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.938004360000001, 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048412121000000002, 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75344 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455570+0000", 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455572+0000", 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2749999999999998e-06 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455578+0000", 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455582+0000", 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4789999999999998e-06 2024-09-16T13:18:36.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503719+0000", 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048137054999999998 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503819+0000", 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9938000000000004e-05 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503982+0000", 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016225500000000001 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75345 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 917504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937987694, 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048702503000000001, 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75345 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455587+0000", 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455589+0000", 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455594+0000", 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6840000000000001e-06 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455598+0000", 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7009999999999998e-06 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504027+0000", 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048429382999999999 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504123+0000", 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5251e-05 2024-09-16T13:18:36.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504289+0000", 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016599200000000001 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75346 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1212416~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937972337000001, 2024-09-16T13:18:36.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048896186000000001, 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75346 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455602+0000", 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2799999999999996e-07 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455604+0000", 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:36.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455609+0000", 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455615+0000", 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2020000000000003e-06 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504315+0000", 2024-09-16T13:18:36.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048700565000000001 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504382+0000", 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7088999999999994e-05 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504498+0000", 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115408 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75347 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1540096~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937953790999998, 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049121102999999999, 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75347 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7100000000000002e-07 2024-09-16T13:18:36.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455623+0000", 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455629+0000", 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455633+0000", 2024-09-16T13:18:36.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9400000000000004e-06 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504546+0000", 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048912718000000001 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504623+0000", 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7726999999999996e-05 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504741+0000", 2024-09-16T13:18:36.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117874 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75353 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 98304~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937818047, 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048929180000000003, 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.434 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75353 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455758+0000", 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8789999999999999e-06 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455763+0000", 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2190000000000003e-06 2024-09-16T13:18:36.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455767+0000", 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504529+0000", 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048761566999999999 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504587+0000", 2024-09-16T13:18:36.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8564999999999999e-05 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504685+0000", 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7754e-05 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75354 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 393216~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:36.438 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937782977000001, 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049086520000000002, 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75354 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455794+0000", 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.508e-06 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455801+0000", 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5630000000000003e-06 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455805+0000", 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504704+0000", 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048898981000000001 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504759+0000", 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4330000000000003e-05 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504877+0000", 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118608 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.442 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75355 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937764345000001, 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118086999999998, 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75355 2024-09-16T13:18:36.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455810+0000", 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455812+0000", 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455818+0000", 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455821+0000", 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504914+0000", 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049092961999999997 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504928+0000", 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3210000000000001e-05 2024-09-16T13:18:36.446 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75356 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937748584000001, 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049129843999999999, 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.447 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75356 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455826+0000", 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0399999999999996e-07 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455828+0000", 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-06 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455835+0000", 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8410000000000002e-06 2024-09-16T13:18:36.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455839+0000", 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2400000000000001e-06 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504947+0000", 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049108546000000003 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504955+0000", 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6109999999999996e-06 2024-09-16T13:18:36.450 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75357 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937731320000001, 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049141246, 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.451 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75357 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455843+0000", 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999998e-07 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455845+0000", 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999998e-06 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455852+0000", 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:18:36.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455856+0000", 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6059999999999999e-06 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504974+0000", 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118385000000001 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504984+0000", 2024-09-16T13:18:36.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7880000000000001e-06 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75358 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937714370999998, 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049149627000000001, 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75358 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455860+0000", 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455862+0000", 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455867+0000", 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.395e-06 2024-09-16T13:18:36.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455873+0000", 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0120000000000002e-06 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505002+0000", 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049128988999999998 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505009+0000", 2024-09-16T13:18:36.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7950000000000002e-06 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75370 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937696618, 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049174830000000003, 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.459 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75370 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455878+0000", 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455880+0000", 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455885+0000", 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:36.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455889+0000", 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6090000000000001e-06 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505040+0000", 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049151242999999997 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505052+0000", 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2055999999999999e-05 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75371 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937681481999999, 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049181864999999998, 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.463 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75371 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455893+0000", 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455895+0000", 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455900+0000", 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.536e-06 2024-09-16T13:18:36.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455904+0000", 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505067+0000", 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049163224999999998 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505074+0000", 2024-09-16T13:18:36.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75372 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937666344, 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049214870000000001, 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.467 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75372 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455908+0000", 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.89e-07 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455910+0000", 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455915+0000", 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455919+0000", 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.472e-06 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505109+0000", 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049190747 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505122+0000", 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3009e-05 2024-09-16T13:18:36.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75373 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937651827, 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049224933999999998, 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.471 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75373 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455924+0000", 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455930+0000", 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4929999999999998e-06 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455936+0000", 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4459999999999998e-06 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505140+0000", 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049203510999999998 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505147+0000", 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1670000000000002e-06 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.474 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75374 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937633748, 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049235427999999998, 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75374 2024-09-16T13:18:36.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455946+0000", 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860000000000001e-06 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455953+0000", 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5119999999999997e-06 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455956+0000", 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.579e-06 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505166+0000", 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049209527000000003 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505175+0000", 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.478 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75375 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.937613467999999, 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049243375999999998, 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75375 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.479 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455961+0000", 2024-09-16T13:18:36.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455963+0000", 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729999999999999e-06 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455970+0000", 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2520000000000002e-06 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455975+0000", 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505198+0000", 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049222852999999997 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505204+0000", 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.482 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.483 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.483 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:36.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:36.485 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:36.475+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:36.584 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:36.579+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:36.635 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:36.635 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:36.635 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:36.636 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:36.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:36.686 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:36.681+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:36.749 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:36.749 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:36.749 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:36.749 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:36.749 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:36.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:36.787 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:36.779+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:36.852 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:36.852 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:36.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:36.852 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:36.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.026131761999999, 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:18:36.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:18:36.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:18:36.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82800 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1130496~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.990073667000001, 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050353660000000001, 2024-09-16T13:18:36.859 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82800 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858530+0000", 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858534+0000", 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8229999999999997e-06 2024-09-16T13:18:36.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858547+0000", 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3302000000000001e-05 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858557+0000", 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-05 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908637+0000", 2024-09-16T13:18:36.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050079811000000002 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908731+0000", 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4085000000000005e-05 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908883+0000", 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000151676 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82801 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1212416~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.990037772000001, 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050667642999999998, 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82801 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858566+0000", 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3499999999999995e-07 2024-09-16T13:18:36.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858570+0000", 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8299999999999998e-06 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858590+0000", 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9831e-05 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858597+0000", 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6729999999999992e-06 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908930+0000", 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050332492999999999 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909036+0000", 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010603999999999999 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909233+0000", 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019704100000000001 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82802 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989997475999999, 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050929965000000001, 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.868 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82802 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:36.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858610+0000", 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0820000000000001e-06 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858655+0000", 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4901000000000003e-05 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858668+0000", 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2343e-05 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909278+0000", 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050610005 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909374+0000", 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6399999999999999e-05 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909535+0000", 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016137599999999999 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.872 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82803 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1474560~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989888625000001, 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051051886999999997, 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82803 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858715+0000", 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999995e-07 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858719+0000", 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.732e-06 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858728+0000", 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6730000000000004e-06 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858733+0000", 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.532e-06 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909572+0000", 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050839399 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909647+0000", 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5073000000000004e-05 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909766+0000", 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118805 2024-09-16T13:18:36.876 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82804 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1654784~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989864253, 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051320085000000001, 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.877 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82804 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3399999999999998e-07 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858742+0000", 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4150000000000002e-06 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858748+0000", 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0090000000000004e-06 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858753+0000", 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1649999999999997e-06 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909812+0000", 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051058612000000003 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909907+0000", 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5795e-05 2024-09-16T13:18:36.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910059+0000", 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015135499999999999 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82805 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1736704~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989844336999999, 2024-09-16T13:18:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051834002999999997, 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82805 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858762+0000", 2024-09-16T13:18:36.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4959999999999999e-06 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858768+0000", 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8830000000000003e-06 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858771+0000", 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9319999999999998e-06 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910078+0000", 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051306696999999998 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910163+0000", 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5065e-05 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910593+0000", 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042940799999999999 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.885 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82806 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1916928~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989827556, 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052120149999999997, 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82806 2024-09-16T13:18:36.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858776+0000", 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2600000000000002e-07 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858778+0000", 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3070000000000001e-06 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858784+0000", 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858788+0000", 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8870000000000002e-06 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910643+0000", 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051854355999999997 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910735+0000", 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2454000000000006e-05 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910896+0000", 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016055600000000001 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82807 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1998848~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989808388, 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052345503000000002, 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.890 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82807 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858797+0000", 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858804+0000", 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0079999999999996e-06 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858808+0000", 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910935+0000", 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052126695000000001 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911010+0000", 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4764000000000004e-05 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911140+0000", 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013022500000000001 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82830 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 770048~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.894 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989790378, 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052616234999999997, 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82830 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:36.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858815+0000", 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3130000000000001e-06 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858821+0000", 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7200000000000003e-06 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858838+0000", 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6492000000000001e-05 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911178+0000", 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052340684999999998 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911272+0000", 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4106999999999999e-05 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911429+0000", 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156389 2024-09-16T13:18:36.898 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82831 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989756591000001, 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052803142999999997, 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.899 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82831 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858848+0000", 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.209e-06 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858858+0000", 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0111e-05 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858870+0000", 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1976e-05 2024-09-16T13:18:36.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858876+0000", 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5649999999999998e-06 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911449+0000", 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052573110999999999 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911521+0000", 2024-09-16T13:18:36.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1209000000000002e-05 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911650+0000", 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012896199999999999 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82832 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1032192~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.98971893, 2024-09-16T13:18:36.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053031663999999999, 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82832 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858885+0000", 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6899999999999996e-07 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858889+0000", 2024-09-16T13:18:36.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.087e-06 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858905+0000", 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5744999999999999e-05 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858909+0000", 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7269999999999998e-06 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911673+0000", 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052763697999999998 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911767+0000", 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4054000000000004e-05 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911916+0000", 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148584 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.907 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82833 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1114112~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989685827000001, 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053225354000000002, 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82833 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.908 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858918+0000", 2024-09-16T13:18:36.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858922+0000", 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.371e-06 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858933+0000", 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1076e-05 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858940+0000", 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9840000000000004e-06 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911944+0000", 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053004044 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912021+0000", 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7189999999999998e-05 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912143+0000", 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012160200000000001 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82834 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1294336~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989656074999999, 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053469475000000002, 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.912 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82834 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858948+0000", 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858952+0000", 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858970+0000", 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.889e-05 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858978+0000", 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912172+0000", 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053193670999999998 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912268+0000", 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6100000000000005e-05 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912416+0000", 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014817500000000001 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.916 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82835 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1376256~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989615780999999, 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053671286999999998, 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82835 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858988+0000", 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858992+0000", 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858999+0000", 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859003+0000", 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912456+0000", 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053453374999999997 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912526+0000", 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9237000000000001e-05 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912658+0000", 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013284000000000001 2024-09-16T13:18:36.920 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82836 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1556480~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989595934, 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053937880000000001, 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82836 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859008+0000", 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859010+0000", 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1950000000000002e-06 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859016+0000", 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8429999999999998e-06 2024-09-16T13:18:36.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859019+0000", 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912697+0000", 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053677322 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912796+0000", 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8991000000000001e-05 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912945+0000", 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149207 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82837 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1638400~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:36.925 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989579911, 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054154068999999999, 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82837 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:36.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859024+0000", 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859026+0000", 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859031+0000", 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6690000000000003e-06 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859035+0000", 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6239999999999999e-06 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912981+0000", 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053946409000000001 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913054+0000", 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3052999999999993e-05 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913177+0000", 2024-09-16T13:18:36.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122803 2024-09-16T13:18:36.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.930 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.930 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82838 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1818624~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.930 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:36.931 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989564473, 2024-09-16T13:18:36.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054416554999999998, 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82838 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859041+0000", 2024-09-16T13:18:36.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2139999999999999e-06 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859047+0000", 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6359999999999999e-06 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859051+0000", 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6519999999999999e-06 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913215+0000", 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054164344000000003 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913307+0000", 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2546999999999993e-05 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913455+0000", 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147639 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.935 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82839 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1900544~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989548811000001, 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054634963000000002, 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82839 2024-09-16T13:18:36.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859055+0000", 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859057+0000", 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3910000000000001e-06 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859066+0000", 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5150000000000006e-06 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859072+0000", 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5810000000000002e-06 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913493+0000", 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054420751000000003 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913563+0000", 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9938999999999996e-05 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913689+0000", 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012626299999999999 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82840 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 2080768~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.989521775, 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054884993999999999, 2024-09-16T13:18:36.940 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82840 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859082+0000", 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5799999999999998e-07 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859086+0000", 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7510000000000002e-06 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859097+0000", 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1085e-05 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859104+0000", 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-06 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913718+0000", 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054614085 2024-09-16T13:18:36.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913814+0000", 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5514999999999996e-05 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913966+0000", 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015253 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:36.944 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:36.945 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:36.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:36.947 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:36.937+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:37.046 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:37.041+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:37.093 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:37.093 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:37.093 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:37.093 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:37.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:37.112 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-16T13:18:37.112 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:18:37.112 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-16T13:18:37.112 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T13:18:37.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T13:18:37.114 INFO:tasks.ceph.osd.7:Started 2024-09-16T13:18:37.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:18:37.147 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:37.145+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:37.204 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-16T13:18:37.204 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-16T13:18:37.204 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-16T13:18:37.204 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-16T13:18:37.204 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-16T13:18:37.204 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-16T13:18:37.204 INFO:teuthology.orchestra.run.smithi151.stderr:config diff 2024-09-16T13:18:37.204 INFO:teuthology.orchestra.run.smithi151.stderr:config diff get 2024-09-16T13:18:37.204 INFO:teuthology.orchestra.run.smithi151.stderr:config get 2024-09-16T13:18:37.205 INFO:teuthology.orchestra.run.smithi151.stderr:config help [] 2024-09-16T13:18:37.205 INFO:teuthology.orchestra.run.smithi151.stderr:config set ... 2024-09-16T13:18:37.205 INFO:teuthology.orchestra.run.smithi151.stderr:admin_socket: invalid command 2024-09-16T13:18:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T13:18:37.206 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-16T13:18:37.209 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:37.209 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:37.209 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:37.209 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:37.210 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:37.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:37.248 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:37.241+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:37.319 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:37.319 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:37.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:37.319 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:37.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.319 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.319 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:37.319 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431809893, 2024-09-16T13:18:37.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:37.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:18:37.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431791058000002, 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.325 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:18:37.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431771918999999, 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:18:37.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:18:37.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:18:37.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:18:37.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:18:37.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:18:37.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:18:37.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.335 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.335 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.335 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:37.335 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431754861000002, 2024-09-16T13:18:37.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:18:37.336 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.336 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.336 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.336 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.336 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.336 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:18:37.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.336 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:37.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:37.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:18:37.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:18:37.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:18:37.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:18:37.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431737501000001, 2024-09-16T13:18:37.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:18:37.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431719791999999, 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.347 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:37.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:37.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:18:37.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.351 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431702493, 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.352 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:18:37.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431667395000002, 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.357 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:18:37.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:18:37.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:18:37.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.362 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.363 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.363 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:37.363 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431644240000001, 2024-09-16T13:18:37.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:18:37.363 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.363 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.363 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:18:37.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.364 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:37.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:37.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:37.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:37.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:18:37.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:18:37.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:18:37.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:18:37.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:18:37.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:18:37.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:18:37.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431603221, 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.369 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:18:37.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:18:37.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431580320999998, 2024-09-16T13:18:37.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:37.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:18:37.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.379 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431555272000001, 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:18:37.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431530050999999, 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.385 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:37.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:37.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:18:37.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:18:37.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:18:37.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:18:37.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:18:37.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:18:37.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:18:37.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:18:37.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:18:37.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:18:37.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:18:37.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:18:37.390 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.390 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.390 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431511686, 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.391 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:18:37.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.392 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:37.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:18:37.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:18:37.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:18:37.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:18:37.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:18:37.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:18:37.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:18:37.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:18:37.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:18:37.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:18:37.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:18:37.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:18:37.397 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.397 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.397 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.397 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.398 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:37.398 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431248548999999, 2024-09-16T13:18:37.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:18:37.398 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.398 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.398 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:18:37.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.399 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:37.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:37.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:18:37.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:18:37.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.403 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431223541000001, 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.404 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:37.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:18:37.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:18:37.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:18:37.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:18:37.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:18:37.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:18:37.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:18:37.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.409 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.409 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.409 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:37.409 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.431195208999998, 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:18:37.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.414 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.430906511, 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:18:37.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.430886400999999, 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:18:37.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.421 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:37.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:18:37.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:18:37.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:18:37.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:18:37.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:18:37.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:18:37.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:18:37.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:18:37.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:18:37.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:18:37.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:18:37.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:18:37.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.425 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.426 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:37.426 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:37.426 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.426685476999999, 2024-09-16T13:18:37.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:18:37.426 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:37.426 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:37.426 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:37.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:37.427 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.427 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:18:37.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.427 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:37.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:37.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:37.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:37.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:37.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:37.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:37.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:18:37.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:18:37.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:37.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:18:37.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:18:37.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:37.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:18:37.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.431 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:37.432 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:37.432 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:37.433 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:37.427+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:37.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:37.534 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:37.529+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:37.541 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:37.541 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:37.542 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:37.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:37.636 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:37.629+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:37.646 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:37.646 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:37.646 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:37.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:37.718 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:37.711+0000 7f2353550540 -1 Falling back to public interface 2024-09-16T13:18:37.737 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:37.731+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:37.752 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:37.752 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:37.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:37.752 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:37.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.752 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66873 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4034560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.752 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132717360000001, 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034635435999999999, 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66873 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615785+0000", 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615786+0000", 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615792+0000", 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1900000000000003e-06 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615795+0000", 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9450000000000002e-06 2024-09-16T13:18:37.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650232+0000", 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034437669999999997 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650301+0000", 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8329000000000003e-05 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650420+0000", 2024-09-16T13:18:37.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119132 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66874 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4042752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132684154, 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034827871000000003, 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.757 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66874 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:37.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615820+0000", 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.063e-06 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615826+0000", 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9739999999999999e-06 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615835+0000", 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5920000000000001e-06 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650445+0000", 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034610100999999997 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650512+0000", 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650645+0000", 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013316399999999999 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.761 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66875 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4050944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132663245, 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032338000000003, 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66875 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615839+0000", 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615841+0000", 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9250000000000002e-06 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615846+0000", 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.711e-06 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615851+0000", 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9579999999999998e-06 2024-09-16T13:18:37.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650676+0000", 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034824524000000003 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650758+0000", 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1547999999999999e-05 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650871+0000", 2024-09-16T13:18:37.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113244 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66876 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4059136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132646350000002, 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035239591000000001, 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66876 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615856+0000", 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615858+0000", 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.857e-06 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615863+0000", 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5389999999999998e-06 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615866+0000", 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650899+0000", 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032448000000001 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650973+0000", 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3627000000000005e-05 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651095+0000", 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012239400000000001 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.770 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66877 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4067328~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.13263165, 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035445691000000001, 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66877 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.771 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:18:37.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615872+0000", 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.703e-06 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615877+0000", 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2650000000000004e-06 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615881+0000", 2024-09-16T13:18:37.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.292e-06 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651123+0000", 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035242373 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651198+0000", 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5096000000000001e-05 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651316+0000", 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011757 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66878 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4075520~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132617756999998, 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035647261999999999, 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.775 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66878 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5400000000000002e-07 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615890+0000", 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6189999999999999e-06 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615896+0000", 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6529999999999999e-06 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615900+0000", 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0069999999999999e-06 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651337+0000", 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035437877999999999 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651395+0000", 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7219999999999998e-05 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651531+0000", 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013653100000000001 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.779 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66879 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132598416, 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035838241999999999, 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66879 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615904+0000", 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:18:37.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615906+0000", 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615911+0000", 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6679999999999998e-06 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615914+0000", 2024-09-16T13:18:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.151e-06 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651558+0000", 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035643672000000001 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651624+0000", 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6070999999999998e-05 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651742+0000", 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011729499999999999 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66880 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4091904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132583835999998, 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036042148000000003, 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.784 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66880 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:37.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2099999999999997e-07 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615924+0000", 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615930+0000", 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6250000000000004e-06 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615933+0000", 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3950000000000001e-06 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651773+0000", 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035839579000000003 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651846+0000", 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2602999999999996e-05 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651960+0000", 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114319 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.788 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66881 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4100096~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132562819, 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036242563999999998, 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66881 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615941+0000", 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.72e-06 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615946+0000", 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.259e-06 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615950+0000", 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:37.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651985+0000", 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036035821000000003 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652058+0000", 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2651999999999995e-05 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652181+0000", 2024-09-16T13:18:37.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123428 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66882 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4108288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132548789000001, 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036437628999999999, 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.793 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66882 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7300000000000002e-07 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615955+0000", 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8199999999999999e-06 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615961+0000", 2024-09-16T13:18:37.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2349999999999996e-06 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615965+0000", 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4620000000000001e-06 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652207+0000", 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036241873000000001 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652276+0000", 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9105999999999999e-05 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652391+0000", 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011476000000000001 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.797 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66883 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4116480~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132533565999999, 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036642620000000001, 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66883 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.77e-07 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615970+0000", 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7519999999999999e-06 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615977+0000", 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9349999999999997e-06 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615985+0000", 2024-09-16T13:18:37.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1389999999999995e-06 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652416+0000", 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036430381999999997 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652487+0000", 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1698000000000004e-05 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652611+0000", 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123337 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66884 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4124672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132269139000002, 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036613363000000003, 2024-09-16T13:18:37.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66884 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.16e-07 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616237+0000", 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5719999999999999e-06 2024-09-16T13:18:37.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616245+0000", 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8820000000000001e-06 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616251+0000", 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9630000000000003e-06 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652638+0000", 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036387705999999999 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652719+0000", 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0469999999999994e-05 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652846+0000", 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126954 2024-09-16T13:18:37.806 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66885 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4132864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132246587000001, 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036785929000000002, 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.807 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66885 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0999999999999999e-07 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616258+0000", 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0269999999999998e-06 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616263+0000", 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8760000000000002e-06 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616267+0000", 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652873+0000", 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036606376000000003 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652956+0000", 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2824000000000005e-05 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653041+0000", 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4918000000000002e-05 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.811 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66886 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4141056~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132230157999999, 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078279999999998, 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66886 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616274+0000", 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.73e-06 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616279+0000", 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616283+0000", 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1420000000000002e-06 2024-09-16T13:18:37.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653119+0000", 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036836016999999999 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653204+0000", 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5575999999999995e-05 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653350+0000", 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014569499999999999 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66887 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4149248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132215655, 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037308557999999999, 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.816 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66887 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.84e-07 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616288+0000", 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7099999999999999e-06 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616294+0000", 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4029999999999998e-06 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616297+0000", 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0079999999999998e-06 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653375+0000", 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078883 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653455+0000", 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0034e-05 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653595+0000", 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013913599999999999 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.820 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66888 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4157440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132201284000001, 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537323999999997, 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66888 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616301+0000", 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616302+0000", 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616308+0000", 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616317+0000", 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2229999999999996e-06 2024-09-16T13:18:37.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653620+0000", 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037302572999999999 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653701+0000", 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1771e-05 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653838+0000", 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013640599999999999 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66889 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4165632~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132180756, 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037755019000000001, 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.825 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66889 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2300000000000002e-07 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616323+0000", 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.905e-06 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616329+0000", 2024-09-16T13:18:37.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4639999999999997e-06 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616332+0000", 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0180000000000002e-06 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653862+0000", 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037529973000000001 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653937+0000", 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4954000000000001e-05 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654076+0000", 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013928200000000001 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66890 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4173824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.829 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132165350000001, 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037917073000000003, 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66890 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616337+0000", 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616338+0000", 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616371+0000", 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2806999999999999e-05 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616375+0000", 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.827e-06 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654115+0000", 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.03773986 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654181+0000", 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5652000000000001e-05 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654253+0000", 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2789999999999999e-05 2024-09-16T13:18:37.833 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66891 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4182016~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132117479000001, 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038160044999999997, 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.834 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66891 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616385+0000", 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616387+0000", 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.356e-06 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616393+0000", 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9519999999999999e-06 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616396+0000", 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2959999999999999e-06 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654302+0000", 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037905031999999998 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654395+0000", 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3560000000000005e-05 2024-09-16T13:18:37.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654544+0000", 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014916500000000001 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66892 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4190208~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.132101668000001, 2024-09-16T13:18:37.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038342319999999999, 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66892 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616407+0000", 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4060000000000002e-06 2024-09-16T13:18:37.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616412+0000", 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-06 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616419+0000", 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.037e-06 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654571+0000", 2024-09-16T13:18:37.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038152835000000003 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654625+0000", 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3643000000000001e-05 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654742+0000", 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117348 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:37.842 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:37.843 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:37.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:37.845 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:37.839+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:37.945 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:37.937+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:37.947 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:37.947 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:37.948 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:37.948 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:37.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:38.046 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:38.037+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:38.058 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:38.059 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:38.059 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:38.059 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:38.059 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:38.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:38.147 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:38.143+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:38.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:38.218 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:38.218 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:38.218 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:38.218 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:38.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.218 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64860 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3899392~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650228808000001, 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035719773000000003, 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64860 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:38.219 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.220 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563336+0000", 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563342+0000", 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3240000000000002e-06 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563345+0000", 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.598973+0000", 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035627631999999999 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599006+0000", 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3495999999999999e-05 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599054+0000", 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7460999999999998e-05 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64861 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3907584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650213926999999, 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035815461, 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64861 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563351+0000", 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.854e-06 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563356+0000", 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5940000000000001e-06 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563360+0000", 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4060000000000001e-06 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599076+0000", 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035716171999999997 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599116+0000", 2024-09-16T13:18:38.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9777999999999998e-05 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599164+0000", 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8307999999999999e-05 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64862 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650199334, 2024-09-16T13:18:38.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035954097999999997, 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64862 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563364+0000", 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563365+0000", 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.821e-06 2024-09-16T13:18:38.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563371+0000", 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0769999999999998e-06 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563374+0000", 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3560000000000001e-06 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599187+0000", 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035813153 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599252+0000", 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4873e-05 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599317+0000", 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5438000000000006e-05 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64863 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3923968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650173779999999, 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036051842000000001, 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64863 2024-09-16T13:18:38.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:38.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563392+0000", 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3140000000000002e-06 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563398+0000", 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.066e-06 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563402+0000", 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599339+0000", 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035937678000000001 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599377+0000", 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599441+0000", 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3380999999999995e-05 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64864 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3932160~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650157375999999, 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036187015000000003, 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64864 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:38.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563406+0000", 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563411+0000", 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563417+0000", 2024-09-16T13:18:38.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563421+0000", 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599458+0000", 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036036870999999998 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599515+0000", 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7067000000000001e-05 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599592+0000", 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7457000000000001e-05 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64865 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3940352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:38.240 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650137986000001, 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036265720000000001, 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64865 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563427+0000", 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.001e-06 2024-09-16T13:18:38.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563433+0000", 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5489999999999997e-06 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563436+0000", 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.484e-06 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599613+0000", 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036176815000000001 2024-09-16T13:18:38.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599646+0000", 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3192999999999999e-05 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599690+0000", 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4240000000000003e-05 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64866 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3948544~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650122207999999, 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036355749999999999, 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64866 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563441+0000", 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563443+0000", 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0870000000000001e-06 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563448+0000", 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.062e-06 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563451+0000", 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:18:38.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599712+0000", 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036261007999999997 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599753+0000", 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0476999999999999e-05 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599796+0000", 2024-09-16T13:18:38.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3226000000000003e-05 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64867 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3956736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650107468000002, 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036484356000000003, 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.249 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64867 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563456+0000", 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563458+0000", 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9980000000000002e-06 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563463+0000", 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563466+0000", 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599810+0000", 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036343683000000002 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599864+0000", 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3565e-05 2024-09-16T13:18:38.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599939+0000", 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5889999999999993e-05 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64868 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3964928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650092310000002, 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036573319999999999, 2024-09-16T13:18:38.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64868 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563471+0000", 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563473+0000", 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563479+0000", 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.178e-06 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563483+0000", 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6339999999999999e-06 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599963+0000", 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036480090999999999 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600001+0000", 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8340999999999998e-05 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600044+0000", 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64869 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3973120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.257 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650076336000001, 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036707818000000003, 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64869 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563487+0000", 2024-09-16T13:18:38.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9100000000000003e-07 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563489+0000", 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563494+0000", 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563498+0000", 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.174e-06 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600061+0000", 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036563127000000001 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600119+0000", 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8476000000000001e-05 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600194+0000", 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4869000000000004e-05 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64870 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3981312~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650061294, 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036852528000000002, 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.262 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64870 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:38.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563502+0000", 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563504+0000", 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9479999999999998e-06 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563509+0000", 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563512+0000", 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3239999999999999e-06 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600218+0000", 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036705812999999997 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600280+0000", 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2298000000000006e-05 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600354+0000", 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3343000000000006e-05 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.266 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64871 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3989504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650046587999999, 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036938311000000001, 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64871 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:38.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563518+0000", 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9130000000000001e-06 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563523+0000", 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563529+0000", 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0710000000000002e-06 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036847958 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600411+0000", 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4388999999999999e-05 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600454+0000", 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3411000000000002e-05 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.270 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64872 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3997696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650030081000001, 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037077038999999999, 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64872 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.271 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563533+0000", 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563535+0000", 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1270000000000001e-06 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563540+0000", 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3120000000000001e-06 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563544+0000", 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4699999999999998e-06 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600476+0000", 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036931691000000003 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600543+0000", 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7475e-05 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600610+0000", 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6397999999999997e-05 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64873 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4005888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650015056000001, 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037239282999999998, 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.275 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64873 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:38.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563550+0000", 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563555+0000", 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563558+0000", 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2009999999999999e-06 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600630+0000", 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037071933000000001 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600702+0000", 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1245e-05 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600787+0000", 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5290000000000005e-05 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64874 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4014080~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:38.279 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.650000609999999, 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037394846000000002, 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64874 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:38.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0400000000000002e-07 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563565+0000", 2024-09-16T13:18:38.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1900000000000002e-06 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563570+0000", 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4199999999999998e-06 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563574+0000", 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600812+0000", 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037237600000000003 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4690000000000006e-05 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600957+0000", 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0463999999999994e-05 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.283 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64875 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4022272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.649984815, 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537883000000001, 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64875 2024-09-16T13:18:38.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:38.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.65e-07 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563580+0000", 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563587+0000", 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2589999999999997e-06 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563591+0000", 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600982+0000", 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037390117 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601044+0000", 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2788999999999998e-05 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601116+0000", 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1282e-05 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64876 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4030464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.649967490000002, 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037682571999999998, 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64876 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563596+0000", 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563613+0000", 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7935999999999999e-05 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563620+0000", 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4740000000000002e-06 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563624+0000", 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601141+0000", 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037517170000000002 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601203+0000", 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2187999999999995e-05 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601278+0000", 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64877 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4038656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:38.292 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.649933880999999, 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037813404000000002, 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64877 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:38.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563631+0000", 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1330000000000002e-06 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563637+0000", 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563640+0000", 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601304+0000", 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037664438000000001 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601372+0000", 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7618000000000002e-05 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601442+0000", 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-05 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64878 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4046848~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.649904708000001, 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037956456, 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64878 2024-09-16T13:18:38.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563659+0000", 2024-09-16T13:18:38.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563661+0000", 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-06 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563667+0000", 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3470000000000004e-06 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563671+0000", 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7170000000000002e-06 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601473+0000", 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037801955999999998 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601537+0000", 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3960000000000004e-05 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601614+0000", 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64879 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4055040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 24.649888068999999, 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038485017000000003, 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:38.301 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64879 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:38.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4099999999999998e-07 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563677+0000", 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1299999999999999e-06 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563682+0000", 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:18:38.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563688+0000", 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3329999999999996e-06 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601640+0000", 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037951864000000002 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601705+0000", 2024-09-16T13:18:38.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5573000000000004e-05 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.602160+0000", 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000454411 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:38.305 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:38.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:38.307 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:38.301+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:38.341 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:38.359 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:38.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:18:38.386 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:38.407 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:38.401+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:38.414 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:38.447 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:38.447 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:38.448 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:38.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:38.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:38.515 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:38.515 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:38.509+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:38.520 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:38.524 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:38.594 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:38.594 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:38.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:38.608 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:38.601+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:38.709 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:38.709 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:38.709 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:38.701+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:38.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:38.811 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:38.803+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:38.813 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:38.813 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:38.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:38.911 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:38.903+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:38.919 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:38.919 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:38.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:39.013 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:39.007+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:39.029 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:39.029 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:39.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:39.114 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:39.107+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:39.148 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:39.148 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:39.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:39.215 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:39.209+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:39.254 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:39.254 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:39.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:39.316 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:39.309+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:39.358 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:39.358 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:39.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:39.391 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:18:39.417 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:39.409+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:39.464 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:39.465 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:39.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:39.518 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:39.511+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:39.570 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:39.570 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:39.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:39.619 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:39.613+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:39.720 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:39.717+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:39.723 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:39.723 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:39.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:39.820 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:39.817+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:39.833 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:39.833 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:39.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:39.921 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:39.913+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:39.936 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:39.936 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:39.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:40.021 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:40.015+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:40.042 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:40.043 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:40.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:40.123 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:40.115+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:40.161 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:40.161 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:40.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:40.224 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:40.219+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:40.275 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:40.275 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:40.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:40.325 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:40.317+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:40.388 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:40.388 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:40.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:40.451 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:40.445+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:40.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:40.527 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:40.519+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:40.557 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:40.557 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:40.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:40.627 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:40.621+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:40.657 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:40.663 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:40.663 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:40.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:40.680 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:40.701 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:40.722 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:40.728 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:40.721+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:40.742 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:40.763 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:40.772 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64860 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3899392~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.205093352999999, 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035719773000000003, 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64860 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563336+0000", 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563342+0000", 2024-09-16T13:18:40.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3240000000000002e-06 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563345+0000", 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.598973+0000", 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035627631999999999 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599006+0000", 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3495999999999999e-05 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599054+0000", 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7460999999999998e-05 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64861 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3907584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:40.777 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.205078472, 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035815461, 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64861 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563351+0000", 2024-09-16T13:18:40.779 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.854e-06 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563356+0000", 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5940000000000001e-06 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563360+0000", 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4060000000000001e-06 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599076+0000", 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035716171999999997 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599116+0000", 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9777999999999998e-05 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599164+0000", 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8307999999999999e-05 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.781 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64862 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.205063879000001, 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035954097999999997, 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64862 2024-09-16T13:18:40.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563364+0000", 2024-09-16T13:18:40.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563365+0000", 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.821e-06 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563371+0000", 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0769999999999998e-06 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563374+0000", 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3560000000000001e-06 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599187+0000", 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035813153 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599252+0000", 2024-09-16T13:18:40.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4873e-05 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599317+0000", 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5438000000000006e-05 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64863 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3923968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:40.786 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.205038325, 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036051842000000001, 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64863 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563392+0000", 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3140000000000002e-06 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563398+0000", 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.066e-06 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563402+0000", 2024-09-16T13:18:40.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599339+0000", 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035937678000000001 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599377+0000", 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599441+0000", 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3380999999999995e-05 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64864 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3932160~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.205021921, 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036187015000000003, 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64864 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563406+0000", 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563411+0000", 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:40.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563417+0000", 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563421+0000", 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599458+0000", 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036036870999999998 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599515+0000", 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7067000000000001e-05 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599592+0000", 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7457000000000001e-05 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64865 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3940352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.205002531000002, 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036265720000000001, 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64865 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563427+0000", 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.001e-06 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563433+0000", 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5489999999999997e-06 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563436+0000", 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.484e-06 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599613+0000", 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036176815000000001 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599646+0000", 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3192999999999999e-05 2024-09-16T13:18:40.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599690+0000", 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4240000000000003e-05 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64866 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3948544~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.800 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204986753, 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036355749999999999, 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64866 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563441+0000", 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:40.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563443+0000", 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0870000000000001e-06 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563448+0000", 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.062e-06 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563451+0000", 2024-09-16T13:18:40.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599712+0000", 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036261007999999997 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599753+0000", 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0476999999999999e-05 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599796+0000", 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3226000000000003e-05 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64867 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3956736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204972012999999, 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036484356000000003, 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.805 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64867 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563456+0000", 2024-09-16T13:18:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:18:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563458+0000", 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9980000000000002e-06 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563463+0000", 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563466+0000", 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:18:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599810+0000", 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036343683000000002 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599864+0000", 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3565e-05 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599939+0000", 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5889999999999993e-05 2024-09-16T13:18:40.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64868 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3964928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204956854999999, 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036573319999999999, 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.810 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64868 2024-09-16T13:18:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563471+0000", 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:40.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563473+0000", 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563479+0000", 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.178e-06 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563483+0000", 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6339999999999999e-06 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599963+0000", 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036480090999999999 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600001+0000", 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8340999999999998e-05 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600044+0000", 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64869 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3973120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204940880999999, 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036707818000000003, 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.815 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64869 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563487+0000", 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9100000000000003e-07 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563489+0000", 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:18:40.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563494+0000", 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563498+0000", 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.174e-06 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600061+0000", 2024-09-16T13:18:40.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036563127000000001 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600119+0000", 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8476000000000001e-05 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600194+0000", 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4869000000000004e-05 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64870 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3981312~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204925839000001, 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036852528000000002, 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64870 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563502+0000", 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:18:40.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563504+0000", 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9479999999999998e-06 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563509+0000", 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563512+0000", 2024-09-16T13:18:40.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3239999999999999e-06 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600218+0000", 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036705812999999997 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600280+0000", 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2298000000000006e-05 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600354+0000", 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3343000000000006e-05 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64871 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3989504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204911133, 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036938311000000001, 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.824 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64871 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563518+0000", 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9130000000000001e-06 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563523+0000", 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:18:40.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563529+0000", 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0710000000000002e-06 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036847958 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600411+0000", 2024-09-16T13:18:40.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4388999999999999e-05 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600454+0000", 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3411000000000002e-05 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64872 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3997696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204894626000002, 2024-09-16T13:18:40.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037077038999999999, 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64872 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563533+0000", 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563535+0000", 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1270000000000001e-06 2024-09-16T13:18:40.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563540+0000", 2024-09-16T13:18:40.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3120000000000001e-06 2024-09-16T13:18:40.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563544+0000", 2024-09-16T13:18:40.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4699999999999998e-06 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600476+0000", 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036931691000000003 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600543+0000", 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7475e-05 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600610+0000", 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6397999999999997e-05 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64873 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4005888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204879600999998, 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037239282999999998, 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.834 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64873 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563550+0000", 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563555+0000", 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563558+0000", 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2009999999999999e-06 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600630+0000", 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037071933000000001 2024-09-16T13:18:40.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600702+0000", 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1245e-05 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600787+0000", 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5290000000000005e-05 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64874 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4014080~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204865155, 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037394846000000002, 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64874 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:40.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0400000000000002e-07 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563565+0000", 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1900000000000002e-06 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563570+0000", 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4199999999999998e-06 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563574+0000", 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600812+0000", 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037237600000000003 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4690000000000006e-05 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600957+0000", 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0463999999999994e-05 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64875 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4022272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204849360000001, 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537883000000001, 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.843 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64875 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.65e-07 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563580+0000", 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563587+0000", 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2589999999999997e-06 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563591+0000", 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600982+0000", 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037390117 2024-09-16T13:18:40.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601044+0000", 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2788999999999998e-05 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601116+0000", 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1282e-05 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64876 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4030464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204832034999999, 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037682571999999998, 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64876 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563596+0000", 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-09-16T13:18:40.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563613+0000", 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7935999999999999e-05 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563620+0000", 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4740000000000002e-06 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563624+0000", 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601141+0000", 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037517170000000002 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601203+0000", 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2187999999999995e-05 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601278+0000", 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64877 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4038656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204798426, 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037813404000000002, 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64877 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563631+0000", 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1330000000000002e-06 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563637+0000", 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563640+0000", 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601304+0000", 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037664438000000001 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601372+0000", 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7618000000000002e-05 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601442+0000", 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-05 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.856 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64878 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4046848~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204769252999998, 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037956456, 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64878 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563659+0000", 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563661+0000", 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-06 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563667+0000", 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3470000000000004e-06 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563671+0000", 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7170000000000002e-06 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601473+0000", 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037801955999999998 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601537+0000", 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3960000000000004e-05 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601614+0000", 2024-09-16T13:18:40.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64879 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4055040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.204752614, 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038485017000000003, 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:40.861 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64879 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4099999999999998e-07 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563677+0000", 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1299999999999999e-06 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563682+0000", 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:18:40.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563688+0000", 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3329999999999996e-06 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601640+0000", 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037951864000000002 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601705+0000", 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5573000000000004e-05 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.602160+0000", 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000454411 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:40.865 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:40.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:40.867 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:40.868 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:40.868 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:40.868 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:40.869 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:40.869 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:40.869 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:40.869 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:40.869 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:40.869 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:40.869 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:40.861+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:40.967 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:40.959+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:41.017 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:41.017 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:41.018 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:41.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:41.068 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:41.061+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:41.128 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:41.129 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:41.129 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:41.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:41.182 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:41.177+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:41.237 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 97.225212377999995, 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.238 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:18:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:18:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:18:41.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:18:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:18:41.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:18:41.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 97.224426879000006, 2024-09-16T13:18:41.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:18:41.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 97.223259956999996, 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:18:41.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:18:41.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:41.253 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 89.173916191000004, 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:41.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:18:41.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:18:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:18:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:18:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:18:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 89.173306750999998, 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:18:41.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:18:41.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:41.262 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 88.676022936999999, 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:18:41.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:18:41.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:18:41.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:18:41.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 88.674922120999994, 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.267 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:18:41.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 88.674782480999994, 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:18:41.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:18:41.276 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 88.674281950999998, 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:41.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.280 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 83.679112564999997, 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 73.422698724, 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:18:41.289 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 72.810135617, 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:41.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.293 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 45.972029446999997, 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:41.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:18:41.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 45.607902113999998, 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.298 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:18:41.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:18:41.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:75420 3.4 e88/23 3:2605e3b8:::rbd_object_map.112d9326a5dd.0000000000000026:head v 88'165, mlcod=88'163)", 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 21.729173098, 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055875079999999997, 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503433+0000", 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503451+0000", 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7640999999999999e-05 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503476+0000", 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5599999999999999e-05 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503495+0000", 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.825e-05 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503611+0000", 2024-09-16T13:18:41.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011631799999999999 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503644+0000", 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3476000000000003e-05 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.508954+0000", 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053093799999999998 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.509019+0000", 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5425999999999993e-05 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:82956 3.0 e90/23 3:02b9fb4a:::rbd_data.112d12abb20f.0000000000000085:head v 90'317, mlcod=90'315)", 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 17.097762099000001, 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035271679999999998, 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134843+0000", 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4700000000000001e-07 2024-09-16T13:18:41.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134851+0000", 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7770000000000002e-06 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134863+0000", 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1705000000000001e-05 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134869+0000", 2024-09-16T13:18:41.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6829999999999996e-06 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.134994+0000", 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012497300000000001 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.135038+0000", 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4548000000000001e-05 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.138295+0000", 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0032565070000000001 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:24.138370+0000", 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5228000000000005e-05 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1195 3.4 e101/23 3:24bef2c8:::rbd_data.112da28a57db.0000000000000020:27 v 101'326, mlcod=99'322)", 2024-09-16T13:18:41.311 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.858959472, 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005296243, 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:18:41.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373646+0000", 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7e-07 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373653+0000", 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4000000000000003e-06 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374730+0000", 2024-09-16T13:18:41.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00107654 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374741+0000", 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1474999999999999e-05 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374829+0000", 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7584999999999996e-05 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374854+0000", 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5269000000000002e-05 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378893+0000", 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040386770000000001 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378942+0000", 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.315 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1197 3.4 e101/23 3:26d31b55:::rbd_data.112da28a57db.000000000000000c:27 v 101'328, mlcod=99'322)", 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.8578423989999999, 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046376150000000003, 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374763+0000", 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5600000000000005e-07 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374784+0000", 2024-09-16T13:18:41.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0781999999999999e-05 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374794+0000", 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0558e-05 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374800+0000", 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375389+0000", 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058970500000000005 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375407+0000", 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7801000000000001e-05 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379351+0000", 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039435030000000001 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379400+0000", 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9332000000000001e-05 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:89342 3.0 e101/23 3:05a18eda:::rbd_data.112da28a57db.0000000000000003:head v 101'352, mlcod=96'350)", 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.497125+0000", 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.7354801769999999, 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038246700000000001, 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.320 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497125+0000", 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497125+0000", 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497127+0000", 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8640000000000001e-06 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497143+0000", 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.677e-05 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497181+0000", 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7648000000000003e-05 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497195+0000", 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3774999999999999e-05 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497386+0000", 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00019153700000000001 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497429+0000", 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2743000000000001e-05 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.500872+0000", 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0034430099999999998 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.500949+0000", 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7323000000000005e-05 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.324 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:89341 3.4 e101/23 3:21ab973d:::rbd_data.112da28a57db.0000000000000002:head v 101'329, mlcod=101'326)", 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.7353826059999999, 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005014859, 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497223+0000", 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8599999999999998e-07 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497231+0000", 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6709999999999999e-06 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497245+0000", 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.489e-05 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497253+0000", 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6199999999999999e-06 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498106+0000", 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00085257599999999996 2024-09-16T13:18:41.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498130+0000", 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4776000000000001e-05 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502154+0000", 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004023703 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502237+0000", 2024-09-16T13:18:41.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2936999999999996e-05 2024-09-16T13:18:41.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.329 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.329 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.329 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:41.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:41.331 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:41.323+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:41.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:18:41.431 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:41.423+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:41.455 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:41.455 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:41.455 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:41.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:41.532 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:41.523+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:41.567 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:41.567 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:41.567 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:41.567 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:41.567 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:41.574 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:41.567+0000 7f2353550540 -1 osd.7 81 log_to_monitors true 2024-09-16T13:18:41.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:41.633 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:41.625+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:41.678 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:41.678 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:41.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:41.678 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62278 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357655693000002, 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027907498999999999, 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62278 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.679 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315829+0000", 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.79e-06 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315835+0000", 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.682e-06 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315838+0000", 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0730000000000001e-06 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343640+0000", 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027802384999999999 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343683+0000", 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3058e-05 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343734+0000", 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1104999999999999e-05 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357641132000001, 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:18:41.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.687 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357617166000001, 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357601996, 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:18:41.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:18:41.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357587573, 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.697 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357566328000001, 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.702 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:18:41.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357551759, 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:18:41.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:18:41.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:18:41.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357532989999999, 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.711 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:41.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:18:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357518095, 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.716 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:18:41.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357503884, 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:18:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:18:41.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357490514999999, 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:18:41.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:18:41.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357475832999999, 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:18:41.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.733 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357460664000001, 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.737 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357446024000001, 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.738 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:18:41.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:18:41.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357415295999999, 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:18:41.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:18:41.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:18:41.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:18:41.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357394279000001, 2024-09-16T13:18:41.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.751 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357374171, 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.752 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:18:41.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.756 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357354268000002, 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:41.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:41.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357339055000001, 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:18:41.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:18:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.357323521000001, 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:41.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:41.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:41.770 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:41.771 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:41.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:41.773 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:41.763+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:41.873 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:41.867+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:41.884 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:41.884 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:41.884 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:41.884 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:41.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:41.975 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:41.967+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:41.999 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:41.999 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:42.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:42.075 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:42.073+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:42.109 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.895021104000001, 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:42.110 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:18:42.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:18:42.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75320 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1114112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.649781378, 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048391086999999999, 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75320 2024-09-16T13:18:42.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455024+0000", 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455026+0000", 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455033+0000", 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8630000000000002e-06 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455037+0000", 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.608e-06 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503178+0000", 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048141646000000003 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503284+0000", 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105917 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503415+0000", 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130359 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75321 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1441792~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.649764386000001, 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048644984000000002, 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.120 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75321 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:42.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455041+0000", 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455043+0000", 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0480000000000001e-06 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455048+0000", 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455052+0000", 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000002e-06 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503461+0000", 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048409079000000001 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503580+0000", 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119234 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503685+0000", 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105006 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75322 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 2064384~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.649738988999999, 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048835044000000001, 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75322 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2599999999999998e-07 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455068+0000", 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.092e-06 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455074+0000", 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.925e-06 2024-09-16T13:18:42.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455078+0000", 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503740+0000", 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048661514000000003 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503822+0000", 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2196999999999999e-05 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503901+0000", 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9091000000000006e-05 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75344 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 589824~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:42.129 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.649235349000001, 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048412121000000002, 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75344 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455570+0000", 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455572+0000", 2024-09-16T13:18:42.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2749999999999998e-06 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455578+0000", 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455582+0000", 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4789999999999998e-06 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503719+0000", 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048137054999999998 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503819+0000", 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9938000000000004e-05 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503982+0000", 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016225500000000001 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.133 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75345 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 917504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.649218683000001, 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048702503000000001, 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75345 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.134 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455587+0000", 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:18:42.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455589+0000", 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455594+0000", 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6840000000000001e-06 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455598+0000", 2024-09-16T13:18:42.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7009999999999998e-06 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504027+0000", 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048429382999999999 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504123+0000", 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5251e-05 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504289+0000", 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016599200000000001 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75346 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1212416~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.649203325999999, 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048896186000000001, 2024-09-16T13:18:42.138 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.139 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.139 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.139 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.139 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.139 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75346 2024-09-16T13:18:42.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.139 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455602+0000", 2024-09-16T13:18:42.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2799999999999996e-07 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455604+0000", 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455609+0000", 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455615+0000", 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2020000000000003e-06 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504315+0000", 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048700565000000001 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504382+0000", 2024-09-16T13:18:42.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7088999999999994e-05 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504498+0000", 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115408 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75347 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1540096~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.649184779999999, 2024-09-16T13:18:42.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049121102999999999, 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75347 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7100000000000002e-07 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455623+0000", 2024-09-16T13:18:42.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455629+0000", 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455633+0000", 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9400000000000004e-06 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504546+0000", 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048912718000000001 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504623+0000", 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7726999999999996e-05 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504741+0000", 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117874 2024-09-16T13:18:42.147 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75353 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 98304~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.649049036000001, 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048929180000000003, 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.148 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75353 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:42.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455758+0000", 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8789999999999999e-06 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455763+0000", 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2190000000000003e-06 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455767+0000", 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504529+0000", 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048761566999999999 2024-09-16T13:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504587+0000", 2024-09-16T13:18:42.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8564999999999999e-05 2024-09-16T13:18:42.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504685+0000", 2024-09-16T13:18:42.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7754e-05 2024-09-16T13:18:42.152 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75354 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 393216~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.649013965999998, 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049086520000000002, 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.153 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75354 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455794+0000", 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.508e-06 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455801+0000", 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5630000000000003e-06 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455805+0000", 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504704+0000", 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048898981000000001 2024-09-16T13:18:42.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504759+0000", 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4330000000000003e-05 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504877+0000", 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118608 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75355 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648995333999999, 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118086999999998, 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.158 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75355 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455810+0000", 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455812+0000", 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455818+0000", 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:18:42.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455821+0000", 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504914+0000", 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049092961999999997 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504928+0000", 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3210000000000001e-05 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75356 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648979572999998, 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049129843999999999, 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.162 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75356 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:42.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455826+0000", 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0399999999999996e-07 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455828+0000", 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-06 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455835+0000", 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8410000000000002e-06 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455839+0000", 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2400000000000001e-06 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504947+0000", 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049108546000000003 2024-09-16T13:18:42.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504955+0000", 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6109999999999996e-06 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75357 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648962309000002, 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049141246, 2024-09-16T13:18:42.166 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75357 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455843+0000", 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999998e-07 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455845+0000", 2024-09-16T13:18:42.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999998e-06 2024-09-16T13:18:42.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455852+0000", 2024-09-16T13:18:42.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:18:42.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455856+0000", 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6059999999999999e-06 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504974+0000", 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118385000000001 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504984+0000", 2024-09-16T13:18:42.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7880000000000001e-06 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75358 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648945359999999, 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049149627000000001, 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.172 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.172 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75358 2024-09-16T13:18:42.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.172 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:42.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455860+0000", 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455862+0000", 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455867+0000", 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.395e-06 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455873+0000", 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0120000000000002e-06 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505002+0000", 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049128988999999998 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505009+0000", 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7950000000000002e-06 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75370 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648927607000001, 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049174830000000003, 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75370 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455878+0000", 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455880+0000", 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455885+0000", 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455889+0000", 2024-09-16T13:18:42.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6090000000000001e-06 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505040+0000", 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049151242999999997 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505052+0000", 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2055999999999999e-05 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75371 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648912470999999, 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049181864999999998, 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75371 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455893+0000", 2024-09-16T13:18:42.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455895+0000", 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455900+0000", 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.536e-06 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455904+0000", 2024-09-16T13:18:42.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505067+0000", 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049163224999999998 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505074+0000", 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75372 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648897333000001, 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049214870000000001, 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75372 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455908+0000", 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.89e-07 2024-09-16T13:18:42.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455910+0000", 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455915+0000", 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455919+0000", 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.472e-06 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505109+0000", 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049190747 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505122+0000", 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3009e-05 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.187 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75373 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648882816, 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049224933999999998, 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75373 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.188 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:42.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455924+0000", 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455930+0000", 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4929999999999998e-06 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455936+0000", 2024-09-16T13:18:42.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4459999999999998e-06 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505140+0000", 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049203510999999998 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505147+0000", 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1670000000000002e-06 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75374 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648864737, 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049235427999999998, 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75374 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:42.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455946+0000", 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860000000000001e-06 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455953+0000", 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5119999999999997e-06 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455956+0000", 2024-09-16T13:18:42.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.579e-06 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505166+0000", 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049209527000000003 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505175+0000", 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75375 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648844456999999, 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049243375999999998, 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75375 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455961+0000", 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455963+0000", 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729999999999999e-06 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455970+0000", 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2520000000000002e-06 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455975+0000", 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505198+0000", 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049222852999999997 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505204+0000", 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.200 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.200 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.200 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.200 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:42.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:42.202 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:42.197+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:42.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:18:42.302 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:42.297+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:42.309 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator fragmentation block 2024-09-16T13:18:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:admin_socket: invalid command 2024-09-16T13:18:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T13:18:42.311 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-16T13:18:42.315 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:42.315 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:42.315 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:42.315 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:42.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:42.404 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:42.397+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:42.428 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:42.428 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:42.429 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:18:42.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:42.467 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:42.459+0000 7f2347a93700 -1 osd.7 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T13:18:42.504 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:42.497+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:42.539 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:18:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 41.712347113999996, 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:18:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82800 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1130496~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.676289018999999, 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050353660000000001, 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82800 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858530+0000", 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858534+0000", 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8229999999999997e-06 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858547+0000", 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3302000000000001e-05 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858557+0000", 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-05 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908637+0000", 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050079811000000002 2024-09-16T13:18:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908731+0000", 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4085000000000005e-05 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908883+0000", 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000151676 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.550 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82801 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1212416~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.676253123999999, 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050667642999999998, 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82801 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858566+0000", 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3499999999999995e-07 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858570+0000", 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8299999999999998e-06 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858590+0000", 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9831e-05 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858597+0000", 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6729999999999992e-06 2024-09-16T13:18:42.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908930+0000", 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050332492999999999 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909036+0000", 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010603999999999999 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909233+0000", 2024-09-16T13:18:42.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019704100000000001 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82802 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.676212828000001, 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050929965000000001, 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.555 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82802 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858610+0000", 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0820000000000001e-06 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858655+0000", 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4901000000000003e-05 2024-09-16T13:18:42.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858668+0000", 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2343e-05 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909278+0000", 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050610005 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909374+0000", 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6399999999999999e-05 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909535+0000", 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016137599999999999 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82803 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1474560~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.676103977, 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051051886999999997, 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.559 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82803 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858715+0000", 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999995e-07 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858719+0000", 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.732e-06 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858728+0000", 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6730000000000004e-06 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858733+0000", 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.532e-06 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909572+0000", 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050839399 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909647+0000", 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5073000000000004e-05 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909766+0000", 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118805 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.563 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82804 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1654784~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.676079604999998, 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051320085000000001, 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82804 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3399999999999998e-07 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858742+0000", 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4150000000000002e-06 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858748+0000", 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0090000000000004e-06 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858753+0000", 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1649999999999997e-06 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909812+0000", 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051058612000000003 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909907+0000", 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5795e-05 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910059+0000", 2024-09-16T13:18:42.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015135499999999999 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82805 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1736704~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.676059688999999, 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051834002999999997, 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82805 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:42.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858762+0000", 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4959999999999999e-06 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858768+0000", 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8830000000000003e-06 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858771+0000", 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9319999999999998e-06 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910078+0000", 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051306696999999998 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910163+0000", 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5065e-05 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910593+0000", 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042940799999999999 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.572 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82806 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1916928~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.676042907999999, 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052120149999999997, 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82806 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858776+0000", 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2600000000000002e-07 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858778+0000", 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3070000000000001e-06 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858784+0000", 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858788+0000", 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8870000000000002e-06 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910643+0000", 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051854355999999997 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910735+0000", 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2454000000000006e-05 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910896+0000", 2024-09-16T13:18:42.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016055600000000001 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82807 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1998848~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.676023740000002, 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052345503000000002, 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.577 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82807 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858797+0000", 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858804+0000", 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0079999999999996e-06 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858808+0000", 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910935+0000", 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052126695000000001 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911010+0000", 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4764000000000004e-05 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911140+0000", 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013022500000000001 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82830 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 770048~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.67600573, 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052616234999999997, 2024-09-16T13:18:42.581 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82830 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858815+0000", 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3130000000000001e-06 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858821+0000", 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7200000000000003e-06 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858838+0000", 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6492000000000001e-05 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911178+0000", 2024-09-16T13:18:42.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052340684999999998 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911272+0000", 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4106999999999999e-05 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911429+0000", 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156389 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82831 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.675971943, 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052803142999999997, 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82831 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858848+0000", 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.209e-06 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858858+0000", 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0111e-05 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858870+0000", 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1976e-05 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858876+0000", 2024-09-16T13:18:42.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5649999999999998e-06 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911449+0000", 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052573110999999999 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911521+0000", 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1209000000000002e-05 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911650+0000", 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012896199999999999 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82832 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1032192~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.675934282, 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053031663999999999, 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.590 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82832 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858885+0000", 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6899999999999996e-07 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858889+0000", 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.087e-06 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858905+0000", 2024-09-16T13:18:42.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5744999999999999e-05 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858909+0000", 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7269999999999998e-06 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911673+0000", 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052763697999999998 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911767+0000", 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4054000000000004e-05 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911916+0000", 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148584 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82833 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1114112~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:42.594 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.675901179, 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053225354000000002, 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82833 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858918+0000", 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858922+0000", 2024-09-16T13:18:42.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.371e-06 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858933+0000", 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1076e-05 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858940+0000", 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9840000000000004e-06 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911944+0000", 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053004044 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912021+0000", 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7189999999999998e-05 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912143+0000", 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012160200000000001 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.598 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82834 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1294336~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.675871427000001, 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053469475000000002, 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.599 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82834 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858948+0000", 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858952+0000", 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858970+0000", 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.889e-05 2024-09-16T13:18:42.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858978+0000", 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912172+0000", 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053193670999999998 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912268+0000", 2024-09-16T13:18:42.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6100000000000005e-05 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912416+0000", 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014817500000000001 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82835 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1376256~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.675831132999999, 2024-09-16T13:18:42.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053671286999999998, 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82835 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858988+0000", 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858992+0000", 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:18:42.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858999+0000", 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859003+0000", 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912456+0000", 2024-09-16T13:18:42.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053453374999999997 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912526+0000", 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9237000000000001e-05 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912658+0000", 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013284000000000001 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82836 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1556480~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.675811285999998, 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053937880000000001, 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82836 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859008+0000", 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:18:42.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859010+0000", 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1950000000000002e-06 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859016+0000", 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8429999999999998e-06 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859019+0000", 2024-09-16T13:18:42.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912697+0000", 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053677322 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912796+0000", 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8991000000000001e-05 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912945+0000", 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149207 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82837 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1638400~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.675795263000001, 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054154068999999999, 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.612 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82837 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:42.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859024+0000", 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859026+0000", 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859031+0000", 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6690000000000003e-06 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859035+0000", 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6239999999999999e-06 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912981+0000", 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053946409000000001 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913054+0000", 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3052999999999993e-05 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913177+0000", 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122803 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.616 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82838 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1818624~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.675779824999999, 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054416554999999998, 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82838 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859041+0000", 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2139999999999999e-06 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859047+0000", 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6359999999999999e-06 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859051+0000", 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6519999999999999e-06 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913215+0000", 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054164344000000003 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913307+0000", 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2546999999999993e-05 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913455+0000", 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147639 2024-09-16T13:18:42.620 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82839 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1900544~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.675764163, 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054634963000000002, 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.621 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82839 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859055+0000", 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859057+0000", 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3910000000000001e-06 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859066+0000", 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5150000000000006e-06 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859072+0000", 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5810000000000002e-06 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913493+0000", 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054420751000000003 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913563+0000", 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9938999999999996e-05 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913689+0000", 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012626299999999999 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82840 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 2080768~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.675737127000001, 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054884993999999999, 2024-09-16T13:18:42.625 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82840 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859082+0000", 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5799999999999998e-07 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859086+0000", 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7510000000000002e-06 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859097+0000", 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1085e-05 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859104+0000", 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-06 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913718+0000", 2024-09-16T13:18:42.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054614085 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913814+0000", 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5514999999999996e-05 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913966+0000", 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015253 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.629 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.630 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.630 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:42.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:42.632 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:42.625+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:42.732 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:42.725+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:42.744 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:42.744 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:42.744 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:42.744 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:42.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:42.832 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:42.827+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:42.862 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:42.863 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:42.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:42.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:42.934 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:42.925+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:42.974 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087833033000003, 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.975 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:18:42.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:18:42.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087814197999997, 2024-09-16T13:18:42.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:18:42.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.983 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087795059000001, 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.984 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:18:42.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:18:42.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:42.988 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087778000999997, 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:42.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:18:42.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087760641000003, 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.993 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:18:42.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:18:42.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:42.997 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087742931999998, 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:18:42.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087725632999998, 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.002 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:18:43.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:18:43.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087690535, 2024-09-16T13:18:43.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:18:43.010 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087667379999999, 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:18:43.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.015 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087626360999998, 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:43.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:18:43.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087603461, 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:18:43.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:18:43.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:43.024 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087578411999999, 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:43.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:18:43.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087553190999998, 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.029 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087534826000002, 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.033 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:18:43.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087271688999998, 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.038 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:18:43.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087246681000003, 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.042 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:43.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.046 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.087218348999997, 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:43.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:18:43.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.086929650999998, 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.055 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.086909540999997, 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:43.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:18:43.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.082708617000002, 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:18:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:18:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:18:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:18:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:43.066 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:43.067 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:43.067 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:43.067 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:43.067 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:43.061+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:43.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:43.083 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:43.109 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:43.131 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:43.153 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-16T13:18:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T13:18:43.158 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T13:18:43.167 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:43.159+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:43.179 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:43.179 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:43.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:43.269 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:43.267+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:43.333 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:43.333 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:43.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:43.370 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:43.363+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:43.446 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:43.446 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:43.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:43.503 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:43.497+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:43.555 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:43.555 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:43.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:43.572 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:43.565+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:43.655 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:43.655 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:43.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:43.707 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:43.701+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:43.773 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:43.765+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:43.807 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:43.807 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:43.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:43.874 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:43.869+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:43.916 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:43.916 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:43.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:43.976 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:43.969+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:44.026 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:44.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:44.077 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:44.071+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:44.178 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:44.169+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:44.180 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:44.181 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:44.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:44.279 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:44.273+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:44.292 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:44.292 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:44.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:44.379 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:44.375+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:44.394 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:44.394 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:44.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:44.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:18:44.481 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:44.475+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:44.514 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:44.514 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:44.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:44.581 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:44.573+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:44.623 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:44.624 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:44.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:44.681 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:44.673+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:44.732 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:44.732 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:44.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:44.782 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:44.775+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:44.841 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:44.841 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:44.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:44.883 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:44.881+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:44.951 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:44.952 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:44.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:44.984 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:44.979+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:45.058 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:45.058 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:45.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:45.113 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:45.107+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:45.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:45.160 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:45.160 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:45.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:45.206 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:45.201+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:45.273 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:45.273 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:45.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:45.287 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:45.281+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:45.302 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:45.324 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:45.346 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:45.366 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:45.380 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:45.380 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:45.386 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:45.388 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:45.379+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:45.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:45.413 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:45.435 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:45.435 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:18:45.454 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:45.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:45.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:45.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:45.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:45.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:45.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:45.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:45.457 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-16T13:18:45.489 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:45.483+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:45.510 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:45.510 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623340308000003, 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:18:45.514 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623321472999997, 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.515 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:18:45.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.519 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623302334000002, 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:45.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:18:45.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623285275999997, 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.525 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:18:45.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.529 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623267916000003, 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623250206999998, 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:18:45.534 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:18:45.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623232907999999, 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623197810000001, 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.543 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:18:45.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623174655, 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.548 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:45.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:18:45.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623133635999999, 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.552 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:18:45.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.556 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623110736000001, 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:18:45.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623085687, 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.561 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:18:45.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623060465999998, 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:18:45.565 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:18:45.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.623042101000003, 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.622778963999998, 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.574 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.578 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.622753955999997, 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:18:45.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:18:45.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.622725623999997, 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.583 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.584 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.584 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:18:45.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.584 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:18:45.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:18:45.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.622436925999999, 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.588 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:18:45.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:45.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.592 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.622416815999998, 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:18:45.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:18:45.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.618215892000002, 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:18:45.597 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:18:45.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.601 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.602 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.602 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:45.602 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:45.602 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:45.603 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:45.595+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:45.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:45.704 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:45.695+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:45.724 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:45.725 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:45.725 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:45.725 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:45.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:45.804 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:45.799+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:45.832 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:45.832 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:45.833 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:45.833 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:45.842 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:45.905 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:45.899+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:45.946 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:45.946 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:45.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:45.946 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:45.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.946 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66873 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4034560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.946 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:45.946 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325697241, 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034635435999999999, 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66873 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:45.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615785+0000", 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615786+0000", 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615792+0000", 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1900000000000003e-06 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615795+0000", 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9450000000000002e-06 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650232+0000", 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034437669999999997 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650301+0000", 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8329000000000003e-05 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650420+0000", 2024-09-16T13:18:45.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119132 2024-09-16T13:18:45.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.951 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66874 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4042752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.951 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:45.951 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325664035000003, 2024-09-16T13:18:45.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034827871000000003, 2024-09-16T13:18:45.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66874 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615820+0000", 2024-09-16T13:18:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.063e-06 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615826+0000", 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9739999999999999e-06 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615835+0000", 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5920000000000001e-06 2024-09-16T13:18:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650445+0000", 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034610100999999997 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650512+0000", 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650645+0000", 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013316399999999999 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66875 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4050944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325643126000003, 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032338000000003, 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.956 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66875 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615839+0000", 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615841+0000", 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9250000000000002e-06 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615846+0000", 2024-09-16T13:18:45.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.711e-06 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615851+0000", 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9579999999999998e-06 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650676+0000", 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034824524000000003 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650758+0000", 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1547999999999999e-05 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650871+0000", 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113244 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66876 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4059136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:45.960 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325626231000001, 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035239591000000001, 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66876 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:45.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615856+0000", 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615858+0000", 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.857e-06 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615863+0000", 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5389999999999998e-06 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615866+0000", 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:45.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650899+0000", 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032448000000001 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650973+0000", 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3627000000000005e-05 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651095+0000", 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012239400000000001 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66877 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4067328~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325611531, 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035445691000000001, 2024-09-16T13:18:45.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66877 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615872+0000", 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.703e-06 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615877+0000", 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2650000000000004e-06 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615881+0000", 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.292e-06 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651123+0000", 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035242373 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651198+0000", 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5096000000000001e-05 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651316+0000", 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011757 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66878 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4075520~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325597637999998, 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035647261999999999, 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66878 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5400000000000002e-07 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615890+0000", 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6189999999999999e-06 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615896+0000", 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6529999999999999e-06 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615900+0000", 2024-09-16T13:18:45.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0069999999999999e-06 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651337+0000", 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035437877999999999 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651395+0000", 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7219999999999998e-05 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651531+0000", 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013653100000000001 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66879 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325578297, 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035838241999999999, 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66879 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:45.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615904+0000", 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615906+0000", 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615911+0000", 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6679999999999998e-06 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615914+0000", 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.151e-06 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651558+0000", 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035643672000000001 2024-09-16T13:18:45.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651624+0000", 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6070999999999998e-05 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651742+0000", 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011729499999999999 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66880 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4091904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325563717000001, 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036042148000000003, 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66880 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.979 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2099999999999997e-07 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615924+0000", 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615930+0000", 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6250000000000004e-06 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615933+0000", 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3950000000000001e-06 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651773+0000", 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035839579000000003 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651846+0000", 2024-09-16T13:18:45.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2602999999999996e-05 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651960+0000", 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114319 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66881 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4100096~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.3255427, 2024-09-16T13:18:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036242563999999998, 2024-09-16T13:18:45.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.984 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.984 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.984 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.984 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.984 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66881 2024-09-16T13:18:45.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.984 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:45.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:18:45.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615941+0000", 2024-09-16T13:18:45.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.72e-06 2024-09-16T13:18:45.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615946+0000", 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.259e-06 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615950+0000", 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651985+0000", 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036035821000000003 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652058+0000", 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2651999999999995e-05 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652181+0000", 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123428 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.988 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66882 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4108288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325528669999997, 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036437628999999999, 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66882 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7300000000000002e-07 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615955+0000", 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8199999999999999e-06 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615961+0000", 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2349999999999996e-06 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615965+0000", 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4620000000000001e-06 2024-09-16T13:18:45.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652207+0000", 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036241873000000001 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652276+0000", 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9105999999999999e-05 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652391+0000", 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011476000000000001 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66883 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4116480~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325513446999999, 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036642620000000001, 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.993 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66883 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:45.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.77e-07 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615970+0000", 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7519999999999999e-06 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615977+0000", 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9349999999999997e-06 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615985+0000", 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1389999999999995e-06 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652416+0000", 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036430381999999997 2024-09-16T13:18:45.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652487+0000", 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1698000000000004e-05 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652611+0000", 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123337 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:45.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66884 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4124672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325249020000001, 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036613363000000003, 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66884 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.998 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.16e-07 2024-09-16T13:18:45.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616237+0000", 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5719999999999999e-06 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616245+0000", 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8820000000000001e-06 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616251+0000", 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9630000000000003e-06 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652638+0000", 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036387705999999999 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652719+0000", 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0469999999999994e-05 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652846+0000", 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126954 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66885 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4132864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325226467999997, 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036785929000000002, 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.002 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66885 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0999999999999999e-07 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616258+0000", 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0269999999999998e-06 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616263+0000", 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8760000000000002e-06 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616267+0000", 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652873+0000", 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036606376000000003 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652956+0000", 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2824000000000005e-05 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653041+0000", 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4918000000000002e-05 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.006 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66886 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4141056~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325210038999998, 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078279999999998, 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66886 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616274+0000", 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.73e-06 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616279+0000", 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616283+0000", 2024-09-16T13:18:46.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1420000000000002e-06 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653119+0000", 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036836016999999999 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653204+0000", 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5575999999999995e-05 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653350+0000", 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014569499999999999 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66887 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4149248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325195536000003, 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037308557999999999, 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.011 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66887 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:46.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.84e-07 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616288+0000", 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7099999999999999e-06 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616294+0000", 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4029999999999998e-06 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616297+0000", 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0079999999999998e-06 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653375+0000", 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078883 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653455+0000", 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0034e-05 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653595+0000", 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013913599999999999 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.015 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66888 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4157440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325181164999996, 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537323999999997, 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66888 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616301+0000", 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:46.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616302+0000", 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616308+0000", 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616317+0000", 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2229999999999996e-06 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653620+0000", 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037302572999999999 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653701+0000", 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1771e-05 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653838+0000", 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013640599999999999 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66889 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4165632~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325160637000003, 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037755019000000001, 2024-09-16T13:18:46.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66889 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2300000000000002e-07 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616323+0000", 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.905e-06 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616329+0000", 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4639999999999997e-06 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616332+0000", 2024-09-16T13:18:46.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0180000000000002e-06 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653862+0000", 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037529973000000001 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653937+0000", 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4954000000000001e-05 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654076+0000", 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013928200000000001 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66890 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4173824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325145231, 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037917073000000003, 2024-09-16T13:18:46.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66890 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616337+0000", 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616338+0000", 2024-09-16T13:18:46.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616371+0000", 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2806999999999999e-05 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616375+0000", 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.827e-06 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654115+0000", 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.03773986 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654181+0000", 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5652000000000001e-05 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654253+0000", 2024-09-16T13:18:46.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2789999999999999e-05 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66891 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4182016~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325097360000001, 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038160044999999997, 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.030 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66891 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:46.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616385+0000", 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616387+0000", 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.356e-06 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.032 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616393+0000", 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9519999999999999e-06 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616396+0000", 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2959999999999999e-06 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654302+0000", 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037905031999999998 2024-09-16T13:18:46.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654395+0000", 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3560000000000005e-05 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654544+0000", 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014916500000000001 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.034 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66892 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4190208~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.325081548999997, 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038342319999999999, 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66892 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.035 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616407+0000", 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4060000000000002e-06 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616412+0000", 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-06 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616419+0000", 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.037e-06 2024-09-16T13:18:46.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654571+0000", 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038152835000000003 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654625+0000", 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3643000000000001e-05 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654742+0000", 2024-09-16T13:18:46.038 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117348 2024-09-16T13:18:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.039 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.039 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:46.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:46.041 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:46.033+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:46.141 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:46.139+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:46.147 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:46.147 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:46.147 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:46.147 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:46.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:46.242 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:46.233+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:46.261 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:46.262 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:46.262 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:46.262 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:46.262 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:46.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:46.343 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:46.339+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:46.419 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:46.419 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:46.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64860 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3899392~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851622481, 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035719773000000003, 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64860 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.420 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563336+0000", 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563342+0000", 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3240000000000002e-06 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563345+0000", 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.598973+0000", 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035627631999999999 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599006+0000", 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3495999999999999e-05 2024-09-16T13:18:46.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599054+0000", 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7460999999999998e-05 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64861 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3907584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851607600000001, 2024-09-16T13:18:46.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035815461, 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64861 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563351+0000", 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.854e-06 2024-09-16T13:18:46.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563356+0000", 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5940000000000001e-06 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563360+0000", 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4060000000000001e-06 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599076+0000", 2024-09-16T13:18:46.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035716171999999997 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599116+0000", 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9777999999999998e-05 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599164+0000", 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8307999999999999e-05 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64862 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851593006999998, 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035954097999999997, 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64862 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563364+0000", 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:46.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563365+0000", 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.821e-06 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563371+0000", 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0769999999999998e-06 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563374+0000", 2024-09-16T13:18:46.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3560000000000001e-06 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599187+0000", 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035813153 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599252+0000", 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4873e-05 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599317+0000", 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5438000000000006e-05 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64863 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3923968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851567453000001, 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036051842000000001, 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.433 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64863 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563392+0000", 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3140000000000002e-06 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563398+0000", 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.066e-06 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563402+0000", 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599339+0000", 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035937678000000001 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599377+0000", 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599441+0000", 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3380999999999995e-05 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.437 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64864 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3932160~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851551049000001, 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036187015000000003, 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64864 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563406+0000", 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563411+0000", 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563417+0000", 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563421+0000", 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599458+0000", 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036036870999999998 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599515+0000", 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7067000000000001e-05 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599592+0000", 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7457000000000001e-05 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.441 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64865 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3940352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851531659000003, 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036265720000000001, 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64865 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.442 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563427+0000", 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.001e-06 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563433+0000", 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5489999999999997e-06 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563436+0000", 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.484e-06 2024-09-16T13:18:46.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599613+0000", 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036176815000000001 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599646+0000", 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3192999999999999e-05 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599690+0000", 2024-09-16T13:18:46.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4240000000000003e-05 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64866 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3948544~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851515880999997, 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036355749999999999, 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.446 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64866 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563441+0000", 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563443+0000", 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0870000000000001e-06 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563448+0000", 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.062e-06 2024-09-16T13:18:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563451+0000", 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599712+0000", 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036261007999999997 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599753+0000", 2024-09-16T13:18:46.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0476999999999999e-05 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599796+0000", 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3226000000000003e-05 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64867 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3956736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851501141, 2024-09-16T13:18:46.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036484356000000003, 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64867 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563456+0000", 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563458+0000", 2024-09-16T13:18:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9980000000000002e-06 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563463+0000", 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563466+0000", 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599810+0000", 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036343683000000002 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599864+0000", 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3565e-05 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599939+0000", 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5889999999999993e-05 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64868 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3964928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851485983000003, 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036573319999999999, 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64868 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.455 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563471+0000", 2024-09-16T13:18:46.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563473+0000", 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563479+0000", 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.178e-06 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563483+0000", 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6339999999999999e-06 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599963+0000", 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036480090999999999 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600001+0000", 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8340999999999998e-05 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600044+0000", 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64869 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3973120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851470009000003, 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036707818000000003, 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64869 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563487+0000", 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9100000000000003e-07 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563489+0000", 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:18:46.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563494+0000", 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563498+0000", 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.174e-06 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600061+0000", 2024-09-16T13:18:46.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036563127000000001 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600119+0000", 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8476000000000001e-05 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600194+0000", 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4869000000000004e-05 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64870 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3981312~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851454967000002, 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036852528000000002, 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64870 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563502+0000", 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:18:46.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563504+0000", 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9479999999999998e-06 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563509+0000", 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563512+0000", 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3239999999999999e-06 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600218+0000", 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036705812999999997 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600280+0000", 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2298000000000006e-05 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600354+0000", 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3343000000000006e-05 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64871 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3989504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851440261, 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036938311000000001, 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.468 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64871 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:46.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563518+0000", 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9130000000000001e-06 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.470 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563523+0000", 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563529+0000", 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0710000000000002e-06 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036847958 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600411+0000", 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4388999999999999e-05 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600454+0000", 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3411000000000002e-05 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64872 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3997696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:46.472 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851423754000002, 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037077038999999999, 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64872 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:46.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563533+0000", 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563535+0000", 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1270000000000001e-06 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563540+0000", 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3120000000000001e-06 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563544+0000", 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4699999999999998e-06 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.475 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600476+0000", 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036931691000000003 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600543+0000", 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7475e-05 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600610+0000", 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6397999999999997e-05 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64873 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4005888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851408728999999, 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037239282999999998, 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64873 2024-09-16T13:18:46.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563550+0000", 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563555+0000", 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563558+0000", 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2009999999999999e-06 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600630+0000", 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037071933000000001 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600702+0000", 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1245e-05 2024-09-16T13:18:46.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600787+0000", 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5290000000000005e-05 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64874 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4014080~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851394282999998, 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037394846000000002, 2024-09-16T13:18:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64874 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0400000000000002e-07 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563565+0000", 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1900000000000002e-06 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563570+0000", 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4199999999999998e-06 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563574+0000", 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600812+0000", 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037237600000000003 2024-09-16T13:18:46.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4690000000000006e-05 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600957+0000", 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0463999999999994e-05 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64875 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4022272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851378488000002, 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537883000000001, 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64875 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.486 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.65e-07 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563580+0000", 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:18:46.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563587+0000", 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2589999999999997e-06 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563591+0000", 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600982+0000", 2024-09-16T13:18:46.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037390117 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601044+0000", 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2788999999999998e-05 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601116+0000", 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1282e-05 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64876 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4030464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851361163, 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037682571999999998, 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64876 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563596+0000", 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-09-16T13:18:46.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563613+0000", 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7935999999999999e-05 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563620+0000", 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4740000000000002e-06 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563624+0000", 2024-09-16T13:18:46.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601141+0000", 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037517170000000002 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601203+0000", 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2187999999999995e-05 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.494 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601278+0000", 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64877 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4038656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851327554000001, 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037813404000000002, 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.495 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64877 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563631+0000", 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1330000000000002e-06 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.497 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563637+0000", 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563640+0000", 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601304+0000", 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037664438000000001 2024-09-16T13:18:46.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601372+0000", 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7618000000000002e-05 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601442+0000", 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-05 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.499 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64878 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4046848~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851298380999999, 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037956456, 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64878 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.500 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563659+0000", 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.501 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563661+0000", 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-06 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563667+0000", 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3470000000000004e-06 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563671+0000", 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7170000000000002e-06 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601473+0000", 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037801955999999998 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601537+0000", 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3960000000000004e-05 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601614+0000", 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64879 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4055040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 32.851281741999998, 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038485017000000003, 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:46.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64879 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.505 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4099999999999998e-07 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563677+0000", 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1299999999999999e-06 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563682+0000", 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563688+0000", 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3329999999999996e-06 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601640+0000", 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037951864000000002 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601705+0000", 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5573000000000004e-05 2024-09-16T13:18:46.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.602160+0000", 2024-09-16T13:18:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000454411 2024-09-16T13:18:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.508 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:46.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:46.510 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:46.503+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:46.610 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:46.605+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:46.616 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:46.616 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:46.616 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:46.616 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:46.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:46.712 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:46.703+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:46.771 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:46.771 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:46.771 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:46.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:46.813 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:46.805+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:46.880 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:46.880 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2432 3.9 e31/23 3:97892f8b:::rbd_data.112dbfd5d823.0000000000000034:head v 31'18, mlcod=31'14)", 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 102.86788039, 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038835710000000002, 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007392+0000", 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007394+0000", 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.623e-06 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007408+0000", 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4337e-05 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007442+0000", 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3318999999999997e-05 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007461+0000", 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9522999999999999e-05 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007530+0000", 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8523999999999996e-05 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.007569+0000", 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8788999999999997e-05 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011223+0000", 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0036545089999999998 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.011276+0000", 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2947e-05 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 102.86709489099999, 2024-09-16T13:18:46.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:18:46.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:18:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:18:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:18:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:18:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 102.865927969, 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.890 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:18:46.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 94.816584203000005, 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:18:46.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:18:46.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 94.815974763, 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:18:46.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:18:46.901 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.902 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.903 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 94.318690949000001, 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:18:46.905 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.906 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.907 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 94.317590132999996, 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.908 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.909 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.910 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:18:46.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 94.317450492999996, 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:18:46.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:18:46.916 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 94.316949962999999, 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.917 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:18:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.920 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:46.921 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 89.321780576999998, 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:46.922 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.923 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:18:46.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:18:46.925 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 79.065366736000001, 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.926 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.927 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:18:46.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 78.452803629000002, 2024-09-16T13:18:46.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:18:46.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:18:46.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.614697458999999, 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:18:46.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.250570126, 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:18:46.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:18:46.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.942 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:75420 3.4 e88/23 3:2605e3b8:::rbd_object_map.112d9326a5dd.0000000000000026:head v 88'165, mlcod=88'163)", 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:46.943 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 27.371841109999998, 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055875079999999997, 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503433+0000", 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503451+0000", 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7640999999999999e-05 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503476+0000", 2024-09-16T13:18:46.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5599999999999999e-05 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503495+0000", 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.825e-05 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503611+0000", 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011631799999999999 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503644+0000", 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3476000000000003e-05 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.508954+0000", 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053093799999999998 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.509019+0000", 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5425999999999993e-05 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1195 3.4 e101/23 3:24bef2c8:::rbd_data.112da28a57db.0000000000000020:27 v 101'326, mlcod=99'322)", 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.5016274840000001, 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005296243, 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373646+0000", 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7e-07 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373653+0000", 2024-09-16T13:18:46.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4000000000000003e-06 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374730+0000", 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00107654 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374741+0000", 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1474999999999999e-05 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374829+0000", 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7584999999999996e-05 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374854+0000", 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5269000000000002e-05 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378893+0000", 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040386770000000001 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378942+0000", 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1197 3.4 e101/23 3:26d31b55:::rbd_data.112da28a57db.000000000000000c:27 v 101'328, mlcod=99'322)", 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.5005104110000005, 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046376150000000003, 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.952 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374763+0000", 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5600000000000005e-07 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374784+0000", 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0781999999999999e-05 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374794+0000", 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0558e-05 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374800+0000", 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375389+0000", 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058970500000000005 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375407+0000", 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7801000000000001e-05 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379351+0000", 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039435030000000001 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379400+0000", 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9332000000000001e-05 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.956 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:89341 3.4 e101/23 3:21ab973d:::rbd_data.112da28a57db.0000000000000002:head v 101'329, mlcod=101'326)", 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.3780506179999996, 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005014859, 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497223+0000", 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8599999999999998e-07 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497231+0000", 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6709999999999999e-06 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497245+0000", 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.489e-05 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497253+0000", 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6199999999999999e-06 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498106+0000", 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00085257599999999996 2024-09-16T13:18:46.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498130+0000", 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4776000000000001e-05 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502154+0000", 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004023703 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502237+0000", 2024-09-16T13:18:46.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2936999999999996e-05 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.4.0:2718 3.9 e106/23 3:9702159d:::rbd_data.112d404789fd.000000000000001c:22 v 106'440, mlcod=105'437)", 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 4.4542324740000003, 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063154409999999998, 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421041+0000", 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:18:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421055+0000", 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4579999999999999e-05 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421080+0000", 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4859e-05 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421089+0000", 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8950000000000004e-06 2024-09-16T13:18:46.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421166+0000", 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6892999999999997e-05 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421197+0000", 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1325000000000002e-05 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427304+0000", 2024-09-16T13:18:46.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061062449999999997 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427356+0000", 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2197000000000002e-05 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.4.0:2792 3.9 e110/23 3:938f1173:::rbd_data.112d94eb720f.0000000000000006:20 v 110'460, mlcod=106'457)", 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:18:46.965 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.41966230599999998, 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041377590000000004, 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:18:46.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2300000000000001e-07 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455621+0000", 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0064000000000001e-05 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455658+0000", 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6288000000000002e-05 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455678+0000", 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0812999999999999e-05 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455817+0000", 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013879800000000001 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455886+0000", 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8842000000000003e-05 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.459651+0000", 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037649519999999998 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.459748+0000", 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7379000000000004e-05 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:46.970 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:46.970 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:46.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:46.972 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:46.963+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:47.072 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:47.067+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:47.077 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:47.077 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:47.077 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:47.077 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:47.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:47.173 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:47.165+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:47.188 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:47.189 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:47.189 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:47.189 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:47.189 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:47.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:47.273 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:47.267+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:47.304 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:47.304 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:47.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:47.304 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:47.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.304 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62278 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.304 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:47.304 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983479041000002, 2024-09-16T13:18:47.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027907498999999999, 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62278 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315829+0000", 2024-09-16T13:18:47.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.79e-06 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315835+0000", 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.682e-06 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315838+0000", 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0730000000000001e-06 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343640+0000", 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027802384999999999 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343683+0000", 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3058e-05 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343734+0000", 2024-09-16T13:18:47.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1104999999999999e-05 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983464480000002, 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.309 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:18:47.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.313 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983440514000002, 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:47.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.317 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983425343999997, 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.318 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:18:47.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.322 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983410921000001, 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:47.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:18:47.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:18:47.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:18:47.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983389676000002, 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:18:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:47.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:47.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:18:47.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:18:47.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:18:47.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:18:47.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:18:47.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:18:47.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:18:47.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:18:47.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:18:47.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:18:47.334 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.334 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.334 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.334 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983375107000001, 2024-09-16T13:18:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:18:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:18:47.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.336 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:47.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:47.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:47.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:18:47.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:18:47.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:18:47.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:18:47.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:18:47.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:18:47.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:18:47.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:18:47.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:18:47.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.340 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:18:47.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:18:47.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:18:47.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:18:47.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.341 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.342 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.342 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:47.342 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983356338, 2024-09-16T13:18:47.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:18:47.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.342 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:18:47.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.347 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983341443, 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.348 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:18:47.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:18:47.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:18:47.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:18:47.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:18:47.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.352 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983327232000001, 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.353 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:18:47.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:18:47.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983313862999999, 2024-09-16T13:18:47.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:47.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:18:47.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:18:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:18:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:18:47.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:18:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:18:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:18:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:18:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:18:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:18:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:18:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983299181, 2024-09-16T13:18:47.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:18:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:18:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:18:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:18:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:47.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:18:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:18:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:18:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:18:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:18:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:18:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:18:47.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:18:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:18:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:18:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983284011999999, 2024-09-16T13:18:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:18:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:18:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:18:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:18:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:18:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:18:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:18:47.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:47.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:18:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:18:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.377 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983269372000002, 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.378 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:18:47.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:18:47.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983238643999996, 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.383 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:47.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:18:47.387 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983217627000002, 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.388 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:18:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:18:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:18:47.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.393 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983197519000001, 2024-09-16T13:18:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:18:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.394 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:18:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:47.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:18:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:18:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:18:47.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.399 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983177615999999, 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.400 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:18:47.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:18:47.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:18:47.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:18:47.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:18:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:18:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:18:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:18:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:18:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:18:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:18:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:18:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.405 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983162403000001, 2024-09-16T13:18:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:18:47.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:18:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:47.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:18:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:18:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:18:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:18:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:18:47.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:18:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.983146869000002, 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:47.413 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:18:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:18:47.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:18:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:18:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:18:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:18:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:47.418 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:47.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:18:47.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:47.421 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:47.417+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:47.449 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:18:47.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:47.521 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:47.513+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:47.549 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:47.550 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:47.550 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:47.550 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:47.550 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:47.550 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:47.550 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:47.550 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:47.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:47.598 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:47.619 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:47.622 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:47.615+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:47.639 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:47.664 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:47.664 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:47.664 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:47.664 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:47.664 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:47.665 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:47.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:47.686 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:47.712 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:47.723 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:47.717+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:47.739 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:47.760 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:18:47.771 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:47.771 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:47.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:47.824 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:47.817+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:47.877 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:47.877 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:47.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:47.925 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:47.919+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:47.989 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:47.989 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:47.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:48.026 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:48.021+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:48.099 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:48.099 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:48.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:48.127 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:48.125+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:48.209 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:48.209 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:48.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:48.228 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:48.221+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:48.318 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:48.318 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:48.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:48.329 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:48.323+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:48.427 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:48.427 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:48.429 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:48.421+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:48.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:18:48.494 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:18:48.530 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:48.523+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:48.546 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:48.546 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:48.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:18:48.632 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:48.625+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:48.654 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:48.655 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:48.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:18:48.732 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:48.727+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:48.767 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:48.767 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:48.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:48.833 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:48.827+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:48.887 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:48.888 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:48.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:48.943 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:48.937+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:48.998 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:48.998 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:49.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:49.035 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:49.029+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:49.101 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:49.101 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:49.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:49.135 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:49.127+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:49.207 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:49.207 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:49.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:49.236 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:49.231+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:49.313 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:49.314 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:49.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:49.337 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:49.329+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:49.423 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:49.423 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:49.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:49.474 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:49.469+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:49.538 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:49.531+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:49.578 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:49.578 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:49.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:49.639 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:49.633+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:49.684 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:49.684 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:49.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:49.742 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:49.737+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:49.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:49.801 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:49.802 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:49.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:49.841 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:49.835+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:49.906 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:49.907 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:49.913 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:49.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:49.931 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:49.942 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:49.933+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:49.951 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:49.971 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:49.992 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:50.012 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:50.013 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:50.013 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:50.014 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:50.022 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:50.043 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:50.037+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:50.117 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:50.117 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:50.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:50.117 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:50.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.117 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62278 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.117 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:50.117 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796426036, 2024-09-16T13:18:50.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027907498999999999, 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62278 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315829+0000", 2024-09-16T13:18:50.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.79e-06 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315835+0000", 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.682e-06 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315838+0000", 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0730000000000001e-06 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343640+0000", 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027802384999999999 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343683+0000", 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3058e-05 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343734+0000", 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1104999999999999e-05 2024-09-16T13:18:50.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796411474999999, 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:18:50.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:50.126 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796387508999999, 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.130 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796372339000001, 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:18:50.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:18:50.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796357915999998, 2024-09-16T13:18:50.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:18:50.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796336670999999, 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:18:50.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796322101999998, 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:18:50.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:18:50.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.148 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796303332999997, 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:18:50.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:18:50.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796288437999998, 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:50.154 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.155 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:18:50.157 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796274226999998, 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.158 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.160 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.162 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796260857999997, 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:18:50.165 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.166 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796246175999997, 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.167 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:50.168 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.169 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.171 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796231007000003, 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.172 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:18:50.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:18:50.175 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796216367, 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.176 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:18:50.178 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.179 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.180 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796185639000001, 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.182 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:18:50.184 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796164621999999, 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.185 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:18:50.187 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.188 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.189 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796144513999998, 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:50.190 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.191 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:18:50.193 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796124611000003, 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.194 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.196 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796109397999999, 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.199 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:18:50.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:18:50.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.201 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.201 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:18:50.201 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:18:50.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:18:50.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:18:50.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.202 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.202 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:18:50.202 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:18:50.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.796093863999999, 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.204 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:50.205 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.206 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:50.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:50.208 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:50.208 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:18:50.208 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:18:50.208 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.208 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.208 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.208 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:50.208 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:50.208 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:50.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:50.210 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:50.205+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:50.248 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:50.270 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:18:50.321 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:50.321 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:50.321 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:50.321 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:50.326 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:50.321+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:50.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:50.412 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:50.403+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:50.428 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:50.428 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:50.428 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:50.428 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:50.428 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:50.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:50.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:18:50.512 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:50.507+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:50.545 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:50.545 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:50.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:50.546 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:50.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.546 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:50.546 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:50.546 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.332397876999998, 2024-09-16T13:18:50.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:18:50.546 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.546 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:50.546 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:18:50.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:50.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:18:50.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:18:50.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:50.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:18:50.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:18:50.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75320 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1114112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.087158151000001, 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048391086999999999, 2024-09-16T13:18:50.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75320 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:50.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455024+0000", 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455026+0000", 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455033+0000", 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8630000000000002e-06 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455037+0000", 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.608e-06 2024-09-16T13:18:50.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503178+0000", 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048141646000000003 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503284+0000", 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105917 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503415+0000", 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130359 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75321 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1441792~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.087141159000002, 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048644984000000002, 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.557 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.558 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75321 2024-09-16T13:18:50.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.558 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:50.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455041+0000", 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455043+0000", 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0480000000000001e-06 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455048+0000", 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455052+0000", 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000002e-06 2024-09-16T13:18:50.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503461+0000", 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048409079000000001 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503580+0000", 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119234 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503685+0000", 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105006 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75322 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 2064384~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.087115762, 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048835044000000001, 2024-09-16T13:18:50.562 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.563 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:50.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.563 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75322 2024-09-16T13:18:50.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.563 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2599999999999998e-07 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455068+0000", 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.092e-06 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455074+0000", 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.925e-06 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455078+0000", 2024-09-16T13:18:50.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503740+0000", 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048661514000000003 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503822+0000", 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2196999999999999e-05 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503901+0000", 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9091000000000006e-05 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75344 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 589824~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086612121999998, 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048412121000000002, 2024-09-16T13:18:50.567 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75344 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455570+0000", 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455572+0000", 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2749999999999998e-06 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455578+0000", 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455582+0000", 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4789999999999998e-06 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503719+0000", 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048137054999999998 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503819+0000", 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9938000000000004e-05 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503982+0000", 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016225500000000001 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.571 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75345 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 917504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086595456000001, 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048702503000000001, 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75345 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:50.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455587+0000", 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455589+0000", 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455594+0000", 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6840000000000001e-06 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455598+0000", 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7009999999999998e-06 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504027+0000", 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048429382999999999 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504123+0000", 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5251e-05 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504289+0000", 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016599200000000001 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75346 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1212416~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086580098999999, 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048896186000000001, 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75346 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.577 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455602+0000", 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2799999999999996e-07 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455604+0000", 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455609+0000", 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455615+0000", 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2020000000000003e-06 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504315+0000", 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048700565000000001 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504382+0000", 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7088999999999994e-05 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504498+0000", 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115408 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.581 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75347 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1540096~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086561552999999, 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049121102999999999, 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75347 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7100000000000002e-07 2024-09-16T13:18:50.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455623+0000", 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455629+0000", 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455633+0000", 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9400000000000004e-06 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504546+0000", 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048912718000000001 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504623+0000", 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7726999999999996e-05 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504741+0000", 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117874 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75353 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 98304~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086425809000001, 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048929180000000003, 2024-09-16T13:18:50.586 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75353 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455758+0000", 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8789999999999999e-06 2024-09-16T13:18:50.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455763+0000", 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2190000000000003e-06 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455767+0000", 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504529+0000", 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048761566999999999 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504587+0000", 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8564999999999999e-05 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504685+0000", 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7754e-05 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.590 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75354 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 393216~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086390738999999, 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049086520000000002, 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75354 2024-09-16T13:18:50.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455794+0000", 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.508e-06 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455801+0000", 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5630000000000003e-06 2024-09-16T13:18:50.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455805+0000", 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504704+0000", 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048898981000000001 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:50.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504759+0000", 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4330000000000003e-05 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504877+0000", 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118608 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75355 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:50.595 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086372106999999, 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118086999999998, 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75355 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:50.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455810+0000", 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455812+0000", 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455818+0000", 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455821+0000", 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:18:50.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504914+0000", 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049092961999999997 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504928+0000", 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3210000000000001e-05 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75356 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086356345999999, 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049129843999999999, 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75356 2024-09-16T13:18:50.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455826+0000", 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0399999999999996e-07 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455828+0000", 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-06 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455835+0000", 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8410000000000002e-06 2024-09-16T13:18:50.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455839+0000", 2024-09-16T13:18:50.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2400000000000001e-06 2024-09-16T13:18:50.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504947+0000", 2024-09-16T13:18:50.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049108546000000003 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504955+0000", 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6109999999999996e-06 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75357 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:50.604 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086339081999999, 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049141246, 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75357 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:50.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455843+0000", 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999998e-07 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455845+0000", 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999998e-06 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455852+0000", 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455856+0000", 2024-09-16T13:18:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6059999999999999e-06 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504974+0000", 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118385000000001 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504984+0000", 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7880000000000001e-06 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75358 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086322132999999, 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049149627000000001, 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75358 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455860+0000", 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455862+0000", 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455867+0000", 2024-09-16T13:18:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.395e-06 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455873+0000", 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0120000000000002e-06 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505002+0000", 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049128988999999998 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505009+0000", 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7950000000000002e-06 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75370 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086304380000001, 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049174830000000003, 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75370 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455878+0000", 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455880+0000", 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455885+0000", 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455889+0000", 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6090000000000001e-06 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505040+0000", 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049151242999999997 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505052+0000", 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2055999999999999e-05 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75371 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086289244, 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049181864999999998, 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75371 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455893+0000", 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455895+0000", 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455900+0000", 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.536e-06 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455904+0000", 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505067+0000", 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049163224999999998 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505074+0000", 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75372 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086274106000001, 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049214870000000001, 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75372 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455908+0000", 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.89e-07 2024-09-16T13:18:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455910+0000", 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455915+0000", 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455919+0000", 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.472e-06 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505109+0000", 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049190747 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505122+0000", 2024-09-16T13:18:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3009e-05 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75373 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086259589000001, 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049224933999999998, 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75373 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455924+0000", 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455930+0000", 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4929999999999998e-06 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455936+0000", 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4459999999999998e-06 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505140+0000", 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049203510999999998 2024-09-16T13:18:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505147+0000", 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1670000000000002e-06 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75374 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.086241510000001, 2024-09-16T13:18:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049235427999999998, 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75374 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455946+0000", 2024-09-16T13:18:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860000000000001e-06 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455953+0000", 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5119999999999997e-06 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455956+0000", 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.579e-06 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505166+0000", 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049209527000000003 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505175+0000", 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75375 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.08622123, 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049243375999999998, 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75375 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455961+0000", 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455963+0000", 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729999999999999e-06 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455970+0000", 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2520000000000002e-06 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455975+0000", 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-09-16T13:18:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505198+0000", 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049222852999999997 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505204+0000", 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:50.639 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:50.639 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:50.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:50.641 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:50.631+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:50.741 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:50.733+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:50.796 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:50.796 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:50.796 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:50.796 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:50.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:50.842 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:50.835+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:50.903 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:50.903 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:50.904 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:50.904 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:50.904 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:50.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:50.943 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:50.937+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:51.012 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:51.012 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:51.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.184510951999997, 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.013 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:18:51.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:18:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:18:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:18:51.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:18:51.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:18:51.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:18:51.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:51.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:18:51.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82800 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1130496~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148452856999999, 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050353660000000001, 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82800 2024-09-16T13:18:51.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858530+0000", 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858534+0000", 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8229999999999997e-06 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858547+0000", 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3302000000000001e-05 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858557+0000", 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-05 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908637+0000", 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050079811000000002 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908731+0000", 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4085000000000005e-05 2024-09-16T13:18:51.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908883+0000", 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000151676 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82801 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1212416~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148416961999999, 2024-09-16T13:18:51.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050667642999999998, 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82801 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858566+0000", 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3499999999999995e-07 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858570+0000", 2024-09-16T13:18:51.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8299999999999998e-06 2024-09-16T13:18:51.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858590+0000", 2024-09-16T13:18:51.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9831e-05 2024-09-16T13:18:51.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858597+0000", 2024-09-16T13:18:51.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6729999999999992e-06 2024-09-16T13:18:51.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908930+0000", 2024-09-16T13:18:51.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050332492999999999 2024-09-16T13:18:51.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909036+0000", 2024-09-16T13:18:51.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010603999999999999 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909233+0000", 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019704100000000001 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82802 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148376666000001, 2024-09-16T13:18:51.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050929965000000001, 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82802 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858610+0000", 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0820000000000001e-06 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858655+0000", 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4901000000000003e-05 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858668+0000", 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2343e-05 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909278+0000", 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050610005 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909374+0000", 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6399999999999999e-05 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909535+0000", 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016137599999999999 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82803 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1474560~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148267815000001, 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051051886999999997, 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.035 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82803 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858715+0000", 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999995e-07 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858719+0000", 2024-09-16T13:18:51.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.732e-06 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858728+0000", 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6730000000000004e-06 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858733+0000", 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.532e-06 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909572+0000", 2024-09-16T13:18:51.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050839399 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909647+0000", 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5073000000000004e-05 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909766+0000", 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118805 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82804 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1654784~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148243442999998, 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051320085000000001, 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.040 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82804 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:51.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3399999999999998e-07 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858742+0000", 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4150000000000002e-06 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858748+0000", 2024-09-16T13:18:51.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0090000000000004e-06 2024-09-16T13:18:51.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858753+0000", 2024-09-16T13:18:51.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1649999999999997e-06 2024-09-16T13:18:51.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909812+0000", 2024-09-16T13:18:51.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051058612000000003 2024-09-16T13:18:51.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909907+0000", 2024-09-16T13:18:51.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5795e-05 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910059+0000", 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015135499999999999 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82805 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1736704~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.045 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148223526999999, 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051834002999999997, 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82805 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858762+0000", 2024-09-16T13:18:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4959999999999999e-06 2024-09-16T13:18:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858768+0000", 2024-09-16T13:18:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8830000000000003e-06 2024-09-16T13:18:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858771+0000", 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9319999999999998e-06 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910078+0000", 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051306696999999998 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910163+0000", 2024-09-16T13:18:51.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5065e-05 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910593+0000", 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042940799999999999 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82806 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1916928~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148206746, 2024-09-16T13:18:51.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052120149999999997, 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82806 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858776+0000", 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2600000000000002e-07 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858778+0000", 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3070000000000001e-06 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858784+0000", 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858788+0000", 2024-09-16T13:18:51.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8870000000000002e-06 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910643+0000", 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051854355999999997 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910735+0000", 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2454000000000006e-05 2024-09-16T13:18:51.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910896+0000", 2024-09-16T13:18:51.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016055600000000001 2024-09-16T13:18:51.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.055 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.056 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82807 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1998848~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.056 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:51.056 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148187578000002, 2024-09-16T13:18:51.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052345503000000002, 2024-09-16T13:18:51.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.056 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.056 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82807 2024-09-16T13:18:51.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.057 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:51.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858797+0000", 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:51.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858804+0000", 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0079999999999996e-06 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858808+0000", 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910935+0000", 2024-09-16T13:18:51.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052126695000000001 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911010+0000", 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4764000000000004e-05 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911140+0000", 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013022500000000001 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.060 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.061 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.061 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82830 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 770048~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.061 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:51.061 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148169568, 2024-09-16T13:18:51.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052616234999999997, 2024-09-16T13:18:51.061 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.061 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.061 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82830 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858815+0000", 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3130000000000001e-06 2024-09-16T13:18:51.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858821+0000", 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7200000000000003e-06 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858838+0000", 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6492000000000001e-05 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911178+0000", 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052340684999999998 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911272+0000", 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4106999999999999e-05 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911429+0000", 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156389 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.065 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82831 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148135781000001, 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052803142999999997, 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.066 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82831 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858848+0000", 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.209e-06 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858858+0000", 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0111e-05 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858870+0000", 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1976e-05 2024-09-16T13:18:51.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858876+0000", 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5649999999999998e-06 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911449+0000", 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052573110999999999 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911521+0000", 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1209000000000002e-05 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911650+0000", 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012896199999999999 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.070 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82832 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1032192~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.14809812, 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053031663999999999, 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82832 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858885+0000", 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6899999999999996e-07 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858889+0000", 2024-09-16T13:18:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.087e-06 2024-09-16T13:18:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858905+0000", 2024-09-16T13:18:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5744999999999999e-05 2024-09-16T13:18:51.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858909+0000", 2024-09-16T13:18:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7269999999999998e-06 2024-09-16T13:18:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911673+0000", 2024-09-16T13:18:51.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052763697999999998 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911767+0000", 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4054000000000004e-05 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911916+0000", 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148584 2024-09-16T13:18:51.075 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.076 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.076 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82833 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1114112~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.076 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:51.076 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148065017, 2024-09-16T13:18:51.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053225354000000002, 2024-09-16T13:18:51.076 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.076 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.077 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82833 2024-09-16T13:18:51.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.077 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:51.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858918+0000", 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858922+0000", 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.371e-06 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858933+0000", 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1076e-05 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858940+0000", 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9840000000000004e-06 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911944+0000", 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053004044 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912021+0000", 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7189999999999998e-05 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912143+0000", 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012160200000000001 2024-09-16T13:18:51.080 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82834 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1294336~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.148035265000001, 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053469475000000002, 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.081 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.082 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.082 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82834 2024-09-16T13:18:51.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.082 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:51.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858948+0000", 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858952+0000", 2024-09-16T13:18:51.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:51.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858970+0000", 2024-09-16T13:18:51.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.889e-05 2024-09-16T13:18:51.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858978+0000", 2024-09-16T13:18:51.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:18:51.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912172+0000", 2024-09-16T13:18:51.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053193670999999998 2024-09-16T13:18:51.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912268+0000", 2024-09-16T13:18:51.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6100000000000005e-05 2024-09-16T13:18:51.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912416+0000", 2024-09-16T13:18:51.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014817500000000001 2024-09-16T13:18:51.087 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.087 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.087 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.087 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82835 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1376256~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.087 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:51.087 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.147994970999999, 2024-09-16T13:18:51.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053671286999999998, 2024-09-16T13:18:51.088 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.088 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.088 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.088 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.088 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.088 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82835 2024-09-16T13:18:51.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.088 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:51.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:18:51.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858988+0000", 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858992+0000", 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858999+0000", 2024-09-16T13:18:51.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859003+0000", 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912456+0000", 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053453374999999997 2024-09-16T13:18:51.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912526+0000", 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9237000000000001e-05 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912658+0000", 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013284000000000001 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82836 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1556480~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.147975123999998, 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053937880000000001, 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82836 2024-09-16T13:18:51.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859008+0000", 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859010+0000", 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1950000000000002e-06 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859016+0000", 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8429999999999998e-06 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859019+0000", 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912697+0000", 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053677322 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912796+0000", 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8991000000000001e-05 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912945+0000", 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149207 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.097 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82837 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1638400~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.147959101000001, 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054154068999999999, 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82837 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859024+0000", 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859026+0000", 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859031+0000", 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6690000000000003e-06 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859035+0000", 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6239999999999999e-06 2024-09-16T13:18:51.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912981+0000", 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053946409000000001 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913054+0000", 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3052999999999993e-05 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913177+0000", 2024-09-16T13:18:51.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122803 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82838 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1818624~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.147943663, 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054416554999999998, 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.102 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82838 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859041+0000", 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2139999999999999e-06 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859047+0000", 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6359999999999999e-06 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859051+0000", 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6519999999999999e-06 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913215+0000", 2024-09-16T13:18:51.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054164344000000003 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913307+0000", 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2546999999999993e-05 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913455+0000", 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147639 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.106 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82839 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1900544~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.147928001, 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054634963000000002, 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.107 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82839 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859055+0000", 2024-09-16T13:18:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859057+0000", 2024-09-16T13:18:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3910000000000001e-06 2024-09-16T13:18:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859066+0000", 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5150000000000006e-06 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859072+0000", 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5810000000000002e-06 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913493+0000", 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054420751000000003 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913563+0000", 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9938999999999996e-05 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913689+0000", 2024-09-16T13:18:51.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012626299999999999 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82840 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 2080768~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.147900965000002, 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054884993999999999, 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82840 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859082+0000", 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5799999999999998e-07 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859086+0000", 2024-09-16T13:18:51.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7510000000000002e-06 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859097+0000", 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1085e-05 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859104+0000", 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-06 2024-09-16T13:18:51.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913718+0000", 2024-09-16T13:18:51.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054614085 2024-09-16T13:18:51.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913814+0000", 2024-09-16T13:18:51.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5514999999999996e-05 2024-09-16T13:18:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913966+0000", 2024-09-16T13:18:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015253 2024-09-16T13:18:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.118 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:51.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:51.120 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:51.117+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:51.220 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:51.211+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:51.224 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:51.225 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:51.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:51.321 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:51.313+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:51.336 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:51.336 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:51.336 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:51.336 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:51.336 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:51.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:51.422 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:51.413+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:51.444 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.557010288000001, 2024-09-16T13:18:51.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:18:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:18:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:18:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:18:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.449 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556991453000002, 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:18:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:18:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:18:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:18:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:18:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:18:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:18:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:18:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556972313999999, 2024-09-16T13:18:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:18:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.456 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.460 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556955256000002, 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:18:51.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:18:51.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556937896000001, 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:18:51.465 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:18:51.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.469 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556920187000003, 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:18:51.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556902888000003, 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:18:51.474 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:18:51.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.478 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556867789999998, 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:18:51.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.483 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556844634999997, 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:18:51.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556803616000003, 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:18:51.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:18:51.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556780715999999, 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.493 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:18:51.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:51.497 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556755666999997, 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:18:51.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:18:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:18:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:51.502 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556730446000003, 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:51.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:18:51.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556712081000001, 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.507 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556448944000003, 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:18:51.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:18:51.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:51.516 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556423936000002, 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:18:51.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:18:51.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556395604000002, 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:18:51.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.525 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556106905999997, 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.526 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:18:51.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.556086796000002, 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:18:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.551885872, 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.539 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:51.540 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:18:51.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:18:51.541 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:51.535+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:51.641 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:51.635+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:51.690 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:51.690 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:51.690 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:51.690 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:51.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:18:51.744 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:51.739+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:51.802 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:51.802 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:51.802 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:51.802 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:51.802 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:51.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:18:51.843 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:51.835+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:51.904 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90616 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:head [delete] snapc 21=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.3282086970000009, 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.870938027, 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.905 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90616 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570781+0000", 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.88e-06 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570792+0000", 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0567000000000001e-05 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570816+0000", 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-05 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570850+0000", 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3779999999999998e-05 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570946+0000", 2024-09-16T13:18:51.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6294999999999999e-05 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570958+0000", 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438058+0000", 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86709994199999996 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438302+0000", 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024435100000000002 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438940+0000", 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063796000000000002 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441545+0000", 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026055509999999998 2024-09-16T13:18:51.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441576+0000", 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0247e-05 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441622+0000", 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6047999999999997e-05 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441716+0000", 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4524000000000005e-05 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90621 3.e 3.2a047d2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.3281074569999998, 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.860052445, 2024-09-16T13:18:51.912 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90621 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570881+0000", 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570885+0000", 2024-09-16T13:18:51.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7230000000000004e-06 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570901+0000", 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5756999999999998e-05 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570908+0000", 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2960000000000001e-06 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571040+0000", 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131803 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571069+0000", 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8694e-05 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430917+0000", 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85984788899999998 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430932+0000", 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5129e-05 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90624 3.e 3.2071842e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.3275241930000004, 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85950820299999997, 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.917 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90624 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571465+0000", 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1799999999999999e-06 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571490+0000", 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5320000000000002e-05 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571524+0000", 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4190000000000003e-05 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571543+0000", 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8674000000000001e-05 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571647+0000", 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103803 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571669+0000", 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2354e-05 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430962+0000", 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85929225799999998 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430971+0000", 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4239999999999999e-06 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2774 3.1b e108/107 3:defecafe:::rbd_data.112d404789fd.0000000000000065:22 v 108'603, mlcod=105'600)", 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:44.555903+0000", 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.3430841119999997, 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045491489999999997, 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.922 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555903+0000", 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555903+0000", 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555904+0000", 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3600000000000002e-07 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555914+0000", 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0012e-05 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555929+0000", 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5153e-05 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555953+0000", 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3635e-05 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.556063+0000", 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110585 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.556138+0000", 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4586999999999996e-05 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.560419+0000", 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042815589999999999 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.560452+0000", 2024-09-16T13:18:51.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2781999999999999e-05 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2775 3.1b e108/107 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:23 v 108'605, mlcod=108'601)", 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.3368139580000005, 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069960880000000001, 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562174+0000", 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7000000000000004e-07 2024-09-16T13:18:51.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562181+0000", 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2459999999999997e-06 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562194+0000", 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2945000000000001e-05 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562200+0000", 2024-09-16T13:18:51.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3679999999999998e-06 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562237+0000", 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7039999999999998e-05 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562262+0000", 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4443999999999998e-05 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569096+0000", 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068345039999999999 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569169+0000", 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2770999999999998e-05 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(3.12 109/108 [PushOp(3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22, version: 83'223, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22@83'223, size: 1048576, copy_subset: [0~1048576], clone_subset: {}, snapset: 22=[]:{22=[22]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:18:51.931 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:45.648012+0000", 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.2509750750000004, 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042118279999999999, 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.648012+0000", 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.648012+0000", 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.648013+0000", 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.218e-06 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.649853+0000", 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018400409999999999 2024-09-16T13:18:51.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.649885+0000", 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1755999999999999e-05 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.649898+0000", 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3508e-05 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.650002+0000", 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103131 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.650011+0000", 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6059999999999992e-06 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.652224+0000", 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022125679999999998 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.935 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(3.12 109/108 [PushOp(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22, version: 83'227, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22@83'227, size: 1048576, copy_subset: [0~1048576], clone_subset: {}, snapset: 22=[]:{22=[22]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.2386782939999996, 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056147790000000003, 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:18:51.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660310+0000", 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-06 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662684+0000", 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023738169999999999 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662716+0000", 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2271999999999999e-05 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662753+0000", 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6851999999999999e-05 2024-09-16T13:18:51.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662882+0000", 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012939700000000001 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662892+0000", 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0315e-05 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.665923+0000", 2024-09-16T13:18:51.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030311140000000001 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90825 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch reconnect cookie 140083147222288 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.4479732429999999, 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0248692960000001, 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.940 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90825 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451018+0000", 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8569999999999997e-06 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451032+0000", 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4586999999999999e-05 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451078+0000", 2024-09-16T13:18:51.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6180000000000002e-05 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451101+0000", 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2773e-05 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451261+0000", 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016003600000000001 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451276+0000", 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4387e-05 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459307+0000", 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080310969999999992 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459318+0000", 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1136e-05 2024-09-16T13:18:51.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471312+0000", 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.011993798 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471508+0000", 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019619700000000001 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.475883+0000", 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004375248 2024-09-16T13:18:51.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2793 3.1b e110/107 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 v 110'613, mlcod=108'609)", 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.442810422, 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070642279999999997, 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456178+0000", 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.517e-06 2024-09-16T13:18:51.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456230+0000", 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2323e-05 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456251+0000", 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0619999999999999e-05 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456269+0000", 2024-09-16T13:18:51.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7883000000000001e-05 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456632+0000", 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036355300000000002 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456686+0000", 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3454e-05 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463210+0000", 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065238550000000003 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463241+0000", 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1022999999999998e-05 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90826 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:18:51.950 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.442746165, 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023049584, 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90826 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:18:51.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456244+0000", 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3000000000000002e-06 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456261+0000", 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7036000000000001e-05 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456294+0000", 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456313+0000", 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8876999999999999e-05 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456444+0000", 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013131300000000001 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456462+0000", 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7920000000000001e-05 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459343+0000", 2024-09-16T13:18:51.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028811539999999999 2024-09-16T13:18:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459351+0000", 2024-09-16T13:18:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5870000000000004e-06 2024-09-16T13:18:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472631+0000", 2024-09-16T13:18:51.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0132794970000001 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472737+0000", 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106496 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.473418+0000", 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068136 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.474875+0000", 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001456564 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479175+0000", 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042996450000000004 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479238+0000", 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3610999999999998e-05 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479290+0000", 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2163999999999997e-05 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1617 3.12 e111/110 3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22 v 111'330, mlcod=0'0)", 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.4330163430000002, 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060955660000000002, 2024-09-16T13:18:51.958 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465972+0000", 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2999999999999999e-07 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465984+0000", 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2133999999999999e-05 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465998+0000", 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4184e-05 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466009+0000", 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1371e-05 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467605+0000", 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015953129999999999 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467632+0000", 2024-09-16T13:18:51.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7447e-05 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472017+0000", 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043852589999999999 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472066+0000", 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8928000000000003e-05 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1618 3.12 e111/110 3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22 v 111'332, mlcod=0'0)", 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.4324196870000003, 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00616418, 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:18:51.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466568+0000", 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6000000000000003e-07 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466575+0000", 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8809999999999998e-06 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466899+0000", 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032458399999999999 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466912+0000", 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2767e-05 2024-09-16T13:18:51.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469208+0000", 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022958369999999998 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469231+0000", 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2578e-05 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472690+0000", 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003459443 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472731+0000", 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1430000000000001e-05 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1621 3.12 e111/110 3:4a3469d0:::rbd_data.112d12abb20f.000000000000003a:28 v 111'338, mlcod=111'334)", 2024-09-16T13:18:51.967 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.4105121260000004, 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005747411, 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:18:51.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488476+0000", 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488497+0000", 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1542000000000001e-05 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488508+0000", 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0461e-05 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488514+0000", 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2990000000000002e-06 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488568+0000", 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4559999999999999e-05 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488591+0000", 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2416e-05 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494170+0000", 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055793279999999997 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494222+0000", 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2079000000000001e-05 2024-09-16T13:18:51.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90830 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.4099114830000001, 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054188910000000003, 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90830 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489077+0000", 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.544e-06 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489084+0000", 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5919999999999997e-06 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489111+0000", 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7484999999999999e-05 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489140+0000", 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8466000000000002e-05 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489192+0000", 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1946e-05 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489255+0000", 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3751000000000007e-05 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489369+0000", 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011379000000000001 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494232+0000", 2024-09-16T13:18:51.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048629679999999996 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494426+0000", 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000194258 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494467+0000", 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0339999999999997e-05 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494494+0000", 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7750999999999999e-05 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1622 3.12 e111/110 3:498ca772:::rbd_data.112d12abb20f.0000000000000017:28 v 111'340, mlcod=111'334)", 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.4094924710000001, 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074768710000000004, 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.978 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489503+0000", 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8720000000000002e-06 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489515+0000", 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2191000000000001e-05 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489524+0000", 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6170000000000007e-06 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489592+0000", 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7670999999999996e-05 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489614+0000", 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2112000000000001e-05 2024-09-16T13:18:51.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496931+0000", 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073173739999999998 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496971+0000", 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0247999999999998e-05 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90832 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch unwatch cookie 140083147222288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.404199728, 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055530110000000001, 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90832 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494788+0000", 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3900000000000004e-07 2024-09-16T13:18:51.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494791+0000", 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9050000000000001e-06 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494799+0000", 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3100000000000001e-06 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494804+0000", 2024-09-16T13:18:51.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2170000000000002e-06 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496030+0000", 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012255689999999999 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496148+0000", 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117612 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:51.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496408+0000", 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026060999999999999 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.499840+0000", 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034319670000000002 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500173+0000", 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033327399999999999 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500232+0000", 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8303000000000001e-05 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500340+0000", 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010860499999999999 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.988 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90833 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.496159+0000", 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.4028275380000004, 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043333319999999996, 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90833 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496159+0000", 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496159+0000", 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496161+0000", 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.128e-06 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496168+0000", 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3200000000000002e-06 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496180+0000", 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2136e-05 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496187+0000", 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7340000000000002e-06 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.498566+0000", 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023790600000000001 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.498662+0000", 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6675999999999994e-05 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500381+0000", 2024-09-16T13:18:51.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001718757 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500431+0000", 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500493+0000", 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1472999999999999e-05 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.993 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:91495 3.12 e114/110 3:4a130900:::rbd_header.112d58339818:head v 114'350, mlcod=114'348)", 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:50.506264+0000", 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.392722748, 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044744950000000002, 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506264+0000", 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506264+0000", 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506265+0000", 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3799999999999996e-07 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506272+0000", 2024-09-16T13:18:51.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7290000000000003e-06 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506289+0000", 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6903999999999999e-05 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506294+0000", 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7849999999999997e-06 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506343+0000", 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8389e-05 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506362+0000", 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9012e-05 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.510689+0000", 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043266770000000001 2024-09-16T13:18:51.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.510739+0000", 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0161000000000003e-05 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:91581 3.13 3:ce90e30d:::rbd_data.112d58339818.0000000000000007:head [delete] snapc 1d=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:50.625374+0000", 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.2736131660000001, 2024-09-16T13:18:51.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044093910000000004, 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 91581 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625374+0000", 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:51.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625374+0000", 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625376+0000", 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3499999999999999e-06 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625389+0000", 2024-09-16T13:18:52.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2819999999999999e-05 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625425+0000", 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6393000000000002e-05 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625444+0000", 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8712999999999999e-05 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:52.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625635+0000", 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019069599999999999 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.626916+0000", 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001281108 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.627267+0000", 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035158500000000002 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.628976+0000", 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017089399999999999 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.629685+0000", 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00070867100000000002 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.629739+0000", 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3937000000000002e-05 2024-09-16T13:18:52.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.629783+0000", 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4178000000000003e-05 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:91826 3.11 3:8c261a44:::rbd_data.112d770f4ae2.000000000000000e:head [delete] snapc 1c=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.25241558400000003, 2024-09-16T13:18:52.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077560520000000003, 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 91826 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:52.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646576+0000", 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2699999999999998e-06 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646592+0000", 2024-09-16T13:18:52.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6439999999999998e-05 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646637+0000", 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5120000000000002e-05 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646657+0000", 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9737000000000001e-05 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:52.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646782+0000", 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012466699999999999 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646790+0000", 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6680000000000007e-06 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651149+0000", 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043590959999999998 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651538+0000", 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000389071 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651714+0000", 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017587200000000001 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654242+0000", 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025272659999999998 2024-09-16T13:18:52.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654249+0000", 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.61e-06 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654294+0000", 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4644000000000001e-05 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654327+0000", 2024-09-16T13:18:52.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3590999999999997e-05 2024-09-16T13:18:52.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:52.011 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:52.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:52.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:52.011 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:52.011 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:52.012 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:52.013 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:52.005+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:52.113 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:52.105+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:52.118 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:52.118 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:52.118 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:52.118 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:52.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:52.214 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:52.209+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:52.276 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:52.276 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:52.277 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:52.277 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:52.277 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:52.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:52.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:52.334 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:52.329+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:52.416 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:52.409+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:52.430 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:52.436 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:52.436 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:52.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:52.450 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:52.471 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:52.491 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:52.517 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:52.517 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:52.509+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:52.536 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:52.555 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:52.555 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:52.555 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:52.564 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:18:52.564 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:18:52.564 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-16T13:18:52.564 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:18:52.564 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:18:52.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-16T13:18:52.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:52.581 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:18:52.618 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:52.611+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:52.688 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:52.688 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "" 2024-09-16T13:18:52.688 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:52.693 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:52.693 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:52.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:52.719 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:52.713+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:52.799 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:52.799 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:52.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:52.859 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:52.853+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:52.908 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:52.909 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:52.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:52.920 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:52.913+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:53.018 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:53.019 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:53.021 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:53.013+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:53.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:53.122 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:53.117+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:53.130 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:53.130 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:53.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:53.224 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:53.217+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:53.282 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:53.283 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:53.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:53.325 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:53.317+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:53.386 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:53.386 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:53.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:53.438 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:53.433+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:53.527 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:53.521+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:53.542 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:18:53.549 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:53.549 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:53.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:53.626 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:53.621+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:53.665 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:53.665 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:53.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:53.727 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:53.719+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:53.773 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:53.773 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:53.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:53.827 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:53.819+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:53.884 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:53.885 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:53.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:53.928 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:53.919+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:53.997 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:53.997 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:54.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:54.028 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:54.025+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:54.129 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:54.123+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:54.149 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:54.150 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:54.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:54.229 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:54.223+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:54.264 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:54.264 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:54.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:54.330 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:54.325+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:54.369 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:54.370 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:54.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:54.431 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:54.423+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:54.477 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:54.477 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:54.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:54.533 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:54.525+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:54.542 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:18:54.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:54.597 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:54.597 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:54.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:18:54.633 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:54.627+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:54.709 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:54.709 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:54.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:18:54.736 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:54.729+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:54.741 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:54.762 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:54.787 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:54.805 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:54.824 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:54.824 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:54.827 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:54.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:18:54.837 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:54.831+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:54.849 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:54.869 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:54.889 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:54.891 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:54.891 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:54.892 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:54.892 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:54.892 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:54.892 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:54.892 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:54.892 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:18:54.934 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:54.934 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:54.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:54.934 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:54.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.934 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90616 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:head [delete] snapc 21=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:18:54.934 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 11.358050843000001, 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.870938027, 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90616 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:18:54.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570781+0000", 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.88e-06 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570792+0000", 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0567000000000001e-05 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570816+0000", 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-05 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570850+0000", 2024-09-16T13:18:54.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3779999999999998e-05 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570946+0000", 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6294999999999999e-05 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570958+0000", 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438058+0000", 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86709994199999996 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438302+0000", 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024435100000000002 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438940+0000", 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063796000000000002 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441545+0000", 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026055509999999998 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441576+0000", 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0247e-05 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441622+0000", 2024-09-16T13:18:54.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6047999999999997e-05 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441716+0000", 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4524000000000005e-05 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90621 3.e 3.2a047d2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:18:54.941 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 11.357949603, 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.860052445, 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90621 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:18:54.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:18:54.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570881+0000", 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570885+0000", 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7230000000000004e-06 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570901+0000", 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5756999999999998e-05 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570908+0000", 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2960000000000001e-06 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571040+0000", 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131803 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571069+0000", 2024-09-16T13:18:54.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8694e-05 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430917+0000", 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85984788899999998 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430932+0000", 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5129e-05 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90624 3.e 3.2071842e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 11.357366339, 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85950820299999997, 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90624 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.948 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571465+0000", 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1799999999999999e-06 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571490+0000", 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5320000000000002e-05 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571524+0000", 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4190000000000003e-05 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571543+0000", 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8674000000000001e-05 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571647+0000", 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103803 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571669+0000", 2024-09-16T13:18:54.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2354e-05 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430962+0000", 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85929225799999998 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430971+0000", 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4239999999999999e-06 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.952 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2774 3.1b e108/107 3:defecafe:::rbd_data.112d404789fd.0000000000000065:22 v 108'603, mlcod=105'600)", 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:44.555903+0000", 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.372926258, 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045491489999999997, 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555903+0000", 2024-09-16T13:18:54.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555903+0000", 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555904+0000", 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3600000000000002e-07 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555914+0000", 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0012e-05 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555929+0000", 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5153e-05 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.555953+0000", 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3635e-05 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.556063+0000", 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110585 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.556138+0000", 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4586999999999996e-05 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.560419+0000", 2024-09-16T13:18:54.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042815589999999999 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.560452+0000", 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2781999999999999e-05 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2775 3.1b e108/107 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:23 v 108'605, mlcod=108'601)", 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:18:54.957 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.366656104, 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069960880000000001, 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562174+0000", 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7000000000000004e-07 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562181+0000", 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2459999999999997e-06 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562194+0000", 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2945000000000001e-05 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562200+0000", 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3679999999999998e-06 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562237+0000", 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7039999999999998e-05 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562262+0000", 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4443999999999998e-05 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569096+0000", 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068345039999999999 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569169+0000", 2024-09-16T13:18:54.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2770999999999998e-05 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(3.12 109/108 [PushOp(3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22, version: 83'223, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22@83'223, size: 1048576, copy_subset: [0~1048576], clone_subset: {}, snapset: 22=[]:{22=[22]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:45.648012+0000", 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.2808172209999995, 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042118279999999999, 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.648012+0000", 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.648012+0000", 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.648013+0000", 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.218e-06 2024-09-16T13:18:54.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.649853+0000", 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018400409999999999 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.649885+0000", 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1755999999999999e-05 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.649898+0000", 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3508e-05 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.650002+0000", 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103131 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.650011+0000", 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6059999999999992e-06 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.652224+0000", 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022125679999999998 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(3.12 109/108 [PushOp(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22, version: 83'227, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22@83'227, size: 1048576, copy_subset: [0~1048576], clone_subset: {}, snapset: 22=[]:{22=[22]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.2685204399999996, 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056147790000000003, 2024-09-16T13:18:54.966 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660310+0000", 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-06 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662684+0000", 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023738169999999999 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662716+0000", 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2271999999999999e-05 2024-09-16T13:18:54.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662753+0000", 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6851999999999999e-05 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662882+0000", 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012939700000000001 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:54.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662892+0000", 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0315e-05 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.665923+0000", 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030311140000000001 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90825 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch reconnect cookie 140083147222288 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:18:54.970 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.4778153889999999, 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0248692960000001, 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90825 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:18:54.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451018+0000", 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8569999999999997e-06 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451032+0000", 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4586999999999999e-05 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451078+0000", 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6180000000000002e-05 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451101+0000", 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2773e-05 2024-09-16T13:18:54.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451261+0000", 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016003600000000001 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451276+0000", 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4387e-05 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459307+0000", 2024-09-16T13:18:54.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080310969999999992 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459318+0000", 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1136e-05 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471312+0000", 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.011993798 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471508+0000", 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019619700000000001 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.475883+0000", 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004375248 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.976 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2793 3.1b e110/107 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 v 110'613, mlcod=108'609)", 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.4726525680000009, 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070642279999999997, 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456178+0000", 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.517e-06 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456230+0000", 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2323e-05 2024-09-16T13:18:54.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456251+0000", 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0619999999999999e-05 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456269+0000", 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7883000000000001e-05 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456632+0000", 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036355300000000002 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456686+0000", 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3454e-05 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463210+0000", 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065238550000000003 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463241+0000", 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1022999999999998e-05 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90826 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.4725883109999991, 2024-09-16T13:18:54.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023049584, 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90826 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456244+0000", 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3000000000000002e-06 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456261+0000", 2024-09-16T13:18:54.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7036000000000001e-05 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456294+0000", 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456313+0000", 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8876999999999999e-05 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456444+0000", 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013131300000000001 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456462+0000", 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7920000000000001e-05 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459343+0000", 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028811539999999999 2024-09-16T13:18:54.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459351+0000", 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5870000000000004e-06 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472631+0000", 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0132794970000001 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:54.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472737+0000", 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106496 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.473418+0000", 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068136 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.474875+0000", 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001456564 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479175+0000", 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042996450000000004 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479238+0000", 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3610999999999998e-05 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479290+0000", 2024-09-16T13:18:54.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2163999999999997e-05 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1617 3.12 e111/110 3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22 v 111'330, mlcod=0'0)", 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.4628584890000003, 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060955660000000002, 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465972+0000", 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2999999999999999e-07 2024-09-16T13:18:54.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465984+0000", 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2133999999999999e-05 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465998+0000", 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4184e-05 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466009+0000", 2024-09-16T13:18:54.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1371e-05 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467605+0000", 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015953129999999999 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467632+0000", 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7447e-05 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472017+0000", 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043852589999999999 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472066+0000", 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8928000000000003e-05 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1618 3.12 e111/110 3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22 v 111'332, mlcod=0'0)", 2024-09-16T13:18:54.993 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.4622618330000003, 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00616418, 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466568+0000", 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6000000000000003e-07 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466575+0000", 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8809999999999998e-06 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466899+0000", 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032458399999999999 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466912+0000", 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2767e-05 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469208+0000", 2024-09-16T13:18:54.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022958369999999998 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469231+0000", 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2578e-05 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472690+0000", 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003459443 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:54.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472731+0000", 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1430000000000001e-05 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1621 3.12 e111/110 3:4a3469d0:::rbd_data.112d12abb20f.000000000000003a:28 v 111'338, mlcod=111'334)", 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.4403542720000004, 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005747411, 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:54.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488476+0000", 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:18:54.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488497+0000", 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1542000000000001e-05 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488508+0000", 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0461e-05 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488514+0000", 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2990000000000002e-06 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488568+0000", 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4559999999999999e-05 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488591+0000", 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2416e-05 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494170+0000", 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055793279999999997 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494222+0000", 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2079000000000001e-05 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90830 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:18:55.002 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.4397536290000001, 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054188910000000003, 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90830 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:18:55.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489077+0000", 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.544e-06 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489084+0000", 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5919999999999997e-06 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489111+0000", 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7484999999999999e-05 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489140+0000", 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8466000000000002e-05 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489192+0000", 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1946e-05 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489255+0000", 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3751000000000007e-05 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489369+0000", 2024-09-16T13:18:55.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011379000000000001 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494232+0000", 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048629679999999996 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494426+0000", 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000194258 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494467+0000", 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0339999999999997e-05 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494494+0000", 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7750999999999999e-05 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1622 3.12 e111/110 3:498ca772:::rbd_data.112d12abb20f.0000000000000017:28 v 111'340, mlcod=111'334)", 2024-09-16T13:18:55.008 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.4393346170000001, 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074768710000000004, 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:55.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489503+0000", 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8720000000000002e-06 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489515+0000", 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2191000000000001e-05 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489524+0000", 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6170000000000007e-06 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489592+0000", 2024-09-16T13:18:55.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7670999999999996e-05 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489614+0000", 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2112000000000001e-05 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496931+0000", 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073173739999999998 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496971+0000", 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0247999999999998e-05 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90832 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch unwatch cookie 140083147222288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.434041874, 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055530110000000001, 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:55.013 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90832 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:55.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494788+0000", 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3900000000000004e-07 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494791+0000", 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9050000000000001e-06 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494799+0000", 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3100000000000001e-06 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494804+0000", 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2170000000000002e-06 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496030+0000", 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012255689999999999 2024-09-16T13:18:55.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496148+0000", 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117612 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496408+0000", 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026060999999999999 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:55.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.499840+0000", 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034319670000000002 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500173+0000", 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033327399999999999 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500232+0000", 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8303000000000001e-05 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500340+0000", 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010860499999999999 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90833 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.496159+0000", 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.4326696840000004, 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043333319999999996, 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:55.019 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90833 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496159+0000", 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:55.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496159+0000", 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496161+0000", 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.128e-06 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496168+0000", 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3200000000000002e-06 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496180+0000", 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2136e-05 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496187+0000", 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7340000000000002e-06 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.498566+0000", 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023790600000000001 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.498662+0000", 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6675999999999994e-05 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500381+0000", 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001718757 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500431+0000", 2024-09-16T13:18:55.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0047999999999999e-05 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500493+0000", 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1472999999999999e-05 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.4397.0:91495 3.12 e114/110 3:4a130900:::rbd_header.112d58339818:head v 114'350, mlcod=114'348)", 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:50.506264+0000", 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.4225648939999997, 2024-09-16T13:18:55.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044744950000000002, 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506264+0000", 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506264+0000", 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506265+0000", 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3799999999999996e-07 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506272+0000", 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7290000000000003e-06 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506289+0000", 2024-09-16T13:18:55.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6903999999999999e-05 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506294+0000", 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7849999999999997e-06 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506343+0000", 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8389e-05 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:55.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.506362+0000", 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9012e-05 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.510689+0000", 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043266770000000001 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.510739+0000", 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0161000000000003e-05 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.028 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:91581 3.13 3:ce90e30d:::rbd_data.112d58339818.0000000000000007:head [delete] snapc 1d=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:50.625374+0000", 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3034553119999996, 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044093910000000004, 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 91581 2024-09-16T13:18:55.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625374+0000", 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625374+0000", 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:55.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625376+0000", 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3499999999999999e-06 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625389+0000", 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2819999999999999e-05 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625425+0000", 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6393000000000002e-05 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625444+0000", 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8712999999999999e-05 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.625635+0000", 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019069599999999999 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.626916+0000", 2024-09-16T13:18:55.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001281108 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.627267+0000", 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035158500000000002 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.628976+0000", 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017089399999999999 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.629685+0000", 2024-09-16T13:18:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00070867100000000002 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.629739+0000", 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3937000000000002e-05 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:50.629783+0000", 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4178000000000003e-05 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:91826 3.11 3:8c261a44:::rbd_data.112d770f4ae2.000000000000000e:head [delete] snapc 1c=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.28225773, 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077560520000000003, 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 91826 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.035 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646576+0000", 2024-09-16T13:18:55.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2699999999999998e-06 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646592+0000", 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6439999999999998e-05 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646637+0000", 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5120000000000002e-05 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646657+0000", 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9737000000000001e-05 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646782+0000", 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012466699999999999 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646790+0000", 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6680000000000007e-06 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651149+0000", 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043590959999999998 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651538+0000", 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000389071 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651714+0000", 2024-09-16T13:18:55.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017587200000000001 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654242+0000", 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025272659999999998 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654249+0000", 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.61e-06 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654294+0000", 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4644000000000001e-05 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654327+0000", 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3590999999999997e-05 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:55.041 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:55.043 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:55.035+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:55.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:55.144 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:55.137+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:55.150 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:55.150 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:55.150 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:55.150 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:55.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:55.246 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:55.237+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:55.254 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:55.254 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:55.254 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:55.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:55.347 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:55.343+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:55.367 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66873 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4034560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747581523000001, 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034635435999999999, 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.368 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66873 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615785+0000", 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615786+0000", 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615792+0000", 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1900000000000003e-06 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615795+0000", 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9450000000000002e-06 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650232+0000", 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034437669999999997 2024-09-16T13:18:55.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650301+0000", 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8329000000000003e-05 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650420+0000", 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119132 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66874 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4042752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747548317000003, 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034827871000000003, 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66874 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.373 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:18:55.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615820+0000", 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.063e-06 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615826+0000", 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9739999999999999e-06 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615835+0000", 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5920000000000001e-06 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650445+0000", 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034610100999999997 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650512+0000", 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:55.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650645+0000", 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013316399999999999 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66875 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4050944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747527408000003, 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032338000000003, 2024-09-16T13:18:55.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66875 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615839+0000", 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615841+0000", 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9250000000000002e-06 2024-09-16T13:18:55.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615846+0000", 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.711e-06 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615851+0000", 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9579999999999998e-06 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650676+0000", 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034824524000000003 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650758+0000", 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1547999999999999e-05 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650871+0000", 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113244 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66876 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4059136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747510513000002, 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035239591000000001, 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.382 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66876 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615856+0000", 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615858+0000", 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.857e-06 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615863+0000", 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5389999999999998e-06 2024-09-16T13:18:55.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615866+0000", 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650899+0000", 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032448000000001 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650973+0000", 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3627000000000005e-05 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651095+0000", 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012239400000000001 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66877 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4067328~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.386 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747495813, 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035445691000000001, 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66877 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:18:55.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615872+0000", 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.703e-06 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615877+0000", 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2650000000000004e-06 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615881+0000", 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.292e-06 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651123+0000", 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035242373 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651198+0000", 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5096000000000001e-05 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651316+0000", 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011757 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66878 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4075520~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747481919999998, 2024-09-16T13:18:55.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035647261999999999, 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66878 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5400000000000002e-07 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615890+0000", 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6189999999999999e-06 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615896+0000", 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6529999999999999e-06 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615900+0000", 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0069999999999999e-06 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651337+0000", 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035437877999999999 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651395+0000", 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7219999999999998e-05 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.395 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651531+0000", 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013653100000000001 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66879 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747462579, 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035838241999999999, 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66879 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615904+0000", 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615906+0000", 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615911+0000", 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6679999999999998e-06 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615914+0000", 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.151e-06 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651558+0000", 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035643672000000001 2024-09-16T13:18:55.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651624+0000", 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6070999999999998e-05 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651742+0000", 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011729499999999999 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66880 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4091904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747447999000002, 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036042148000000003, 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66880 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.401 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:18:55.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2099999999999997e-07 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615924+0000", 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615930+0000", 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6250000000000004e-06 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615933+0000", 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3950000000000001e-06 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651773+0000", 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035839579000000003 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651846+0000", 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2602999999999996e-05 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651960+0000", 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114319 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66881 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4100096~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747426982, 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036242563999999998, 2024-09-16T13:18:55.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66881 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615941+0000", 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.72e-06 2024-09-16T13:18:55.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615946+0000", 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.259e-06 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615950+0000", 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651985+0000", 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036035821000000003 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652058+0000", 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2651999999999995e-05 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652181+0000", 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123428 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.409 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66882 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4108288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747412951999998, 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036437628999999999, 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66882 2024-09-16T13:18:55.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7300000000000002e-07 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615955+0000", 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8199999999999999e-06 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615961+0000", 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2349999999999996e-06 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615965+0000", 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4620000000000001e-06 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652207+0000", 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036241873000000001 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652276+0000", 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9105999999999999e-05 2024-09-16T13:18:55.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652391+0000", 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011476000000000001 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66883 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4116480~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747397728999999, 2024-09-16T13:18:55.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036642620000000001, 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66883 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.77e-07 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.416 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615970+0000", 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7519999999999999e-06 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615977+0000", 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9349999999999997e-06 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615985+0000", 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1389999999999995e-06 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652416+0000", 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036430381999999997 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652487+0000", 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1698000000000004e-05 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652611+0000", 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123337 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66884 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4124672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747133302000002, 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036613363000000003, 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66884 2024-09-16T13:18:55.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.16e-07 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616237+0000", 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5719999999999999e-06 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616245+0000", 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8820000000000001e-06 2024-09-16T13:18:55.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616251+0000", 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9630000000000003e-06 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652638+0000", 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036387705999999999 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652719+0000", 2024-09-16T13:18:55.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0469999999999994e-05 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652846+0000", 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126954 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66885 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4132864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:55.423 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747110749999997, 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036785929000000002, 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66885 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0999999999999999e-07 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616258+0000", 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0269999999999998e-06 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616263+0000", 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8760000000000002e-06 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616267+0000", 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:18:55.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652873+0000", 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036606376000000003 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652956+0000", 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2824000000000005e-05 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653041+0000", 2024-09-16T13:18:55.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4918000000000002e-05 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66886 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4141056~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747094320999999, 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078279999999998, 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.428 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66886 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:55.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616274+0000", 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.73e-06 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616279+0000", 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616283+0000", 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1420000000000002e-06 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653119+0000", 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036836016999999999 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653204+0000", 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5575999999999995e-05 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653350+0000", 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014569499999999999 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.432 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66887 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4149248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747079818000003, 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037308557999999999, 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66887 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.84e-07 2024-09-16T13:18:55.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616288+0000", 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7099999999999999e-06 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616294+0000", 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4029999999999998e-06 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616297+0000", 2024-09-16T13:18:55.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0079999999999998e-06 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653375+0000", 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078883 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653455+0000", 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0034e-05 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653595+0000", 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013913599999999999 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66888 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4157440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747065446999997, 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537323999999997, 2024-09-16T13:18:55.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66888 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.438 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616301+0000", 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616302+0000", 2024-09-16T13:18:55.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616308+0000", 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616317+0000", 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2229999999999996e-06 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653620+0000", 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037302572999999999 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653701+0000", 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1771e-05 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653838+0000", 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013640599999999999 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.441 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66889 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4165632~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747044918999997, 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037755019000000001, 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66889 2024-09-16T13:18:55.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2300000000000002e-07 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616323+0000", 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.905e-06 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616329+0000", 2024-09-16T13:18:55.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4639999999999997e-06 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616332+0000", 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0180000000000002e-06 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653862+0000", 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037529973000000001 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653937+0000", 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4954000000000001e-05 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654076+0000", 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013928200000000001 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66890 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4173824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.747029513000001, 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037917073000000003, 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66890 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.447 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616337+0000", 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:55.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616338+0000", 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616371+0000", 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2806999999999999e-05 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616375+0000", 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.827e-06 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654115+0000", 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.03773986 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654181+0000", 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5652000000000001e-05 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654253+0000", 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2789999999999999e-05 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66891 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4182016~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.746981642000002, 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038160044999999997, 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.451 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66891 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616385+0000", 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616387+0000", 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.356e-06 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616393+0000", 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9519999999999999e-06 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616396+0000", 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2959999999999999e-06 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654302+0000", 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037905031999999998 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654395+0000", 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3560000000000005e-05 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654544+0000", 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014916500000000001 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66892 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4190208~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.746965830999997, 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038342319999999999, 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66892 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.456 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:18:55.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616407+0000", 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4060000000000002e-06 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616412+0000", 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-06 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616419+0000", 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.037e-06 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654571+0000", 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038152835000000003 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654625+0000", 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3643000000000001e-05 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654742+0000", 2024-09-16T13:18:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117348 2024-09-16T13:18:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.460 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:55.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:55.462 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:55.459+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:55.562 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:55.557+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:55.579 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:55.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:92539 3.0 3:02df8220:::rbd_header.112d3e228c72:head [watch unwatch cookie 140083112854976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:55.575992+0000", 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.001672354, 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0017039799999999999, 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 92539 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.580 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.575992+0000", 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.575992+0000", 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.581 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.575993+0000", 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0699999999999996e-07 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.575996+0000", 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.044e-06 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576002+0000", 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4189999999999999e-06 2024-09-16T13:18:55.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576008+0000", 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.361e-06 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576313+0000", 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00030539199999999998 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.583 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576386+0000", 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2441999999999995e-05 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for subops from 2", 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576522+0000", 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013645700000000001 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "op_commit", 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.577158+0000", 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00063604700000000004 2024-09-16T13:18:55.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.577520+0000", 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00036188300000000002 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.577581+0000", 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0761000000000003e-05 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.585 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.577711+0000", 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000129935 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:92540 3.0 3:02df8220:::rbd_header.112d3e228c72:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:55.576376+0000", 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0012888089999999999, 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0014508769999999999, 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.586 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "waiting for rw locks", 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 92540 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576376+0000", 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.587 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576376+0000", 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576377+0000", 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1650000000000001e-06 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576382+0000", 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3170000000000001e-06 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576393+0000", 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1093e-05 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576406+0000", 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2577e-05 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.576894+0000", 2024-09-16T13:18:55.589 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00048780200000000002 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for rw locks", 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.577008+0000", 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114473 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 2 2024-09-16T13:18:55.590 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:55.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:55.663 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:55.657+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:55.689 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:55.689 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:55.689 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:55.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:55.764 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:55.759+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:55.801 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:55.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64860 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3899392~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234963501000003, 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035719773000000003, 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64860 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:18:55.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563336+0000", 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563342+0000", 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3240000000000002e-06 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563345+0000", 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.598973+0000", 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035627631999999999 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599006+0000", 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3495999999999999e-05 2024-09-16T13:18:55.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599054+0000", 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7460999999999998e-05 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64861 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3907584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234948619999997, 2024-09-16T13:18:55.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035815461, 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64861 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563351+0000", 2024-09-16T13:18:55.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.854e-06 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563356+0000", 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5940000000000001e-06 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563360+0000", 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4060000000000001e-06 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599076+0000", 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035716171999999997 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599116+0000", 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9777999999999998e-05 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599164+0000", 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8307999999999999e-05 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64862 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234934027000001, 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035954097999999997, 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64862 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.811 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563364+0000", 2024-09-16T13:18:55.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563365+0000", 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.821e-06 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563371+0000", 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0769999999999998e-06 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563374+0000", 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3560000000000001e-06 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599187+0000", 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035813153 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599252+0000", 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4873e-05 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599317+0000", 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5438000000000006e-05 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64863 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3923968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234908472999997, 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036051842000000001, 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64863 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563392+0000", 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3140000000000002e-06 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563398+0000", 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.066e-06 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563402+0000", 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599339+0000", 2024-09-16T13:18:55.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035937678000000001 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599377+0000", 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599441+0000", 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3380999999999995e-05 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64864 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3932160~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234892068999997, 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036187015000000003, 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64864 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.820 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:18:55.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563406+0000", 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563411+0000", 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563417+0000", 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563421+0000", 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599458+0000", 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036036870999999998 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599515+0000", 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7067000000000001e-05 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599592+0000", 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7457000000000001e-05 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64865 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3940352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234872678999999, 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036265720000000001, 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64865 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:18:55.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563427+0000", 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.001e-06 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563433+0000", 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5489999999999997e-06 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563436+0000", 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.484e-06 2024-09-16T13:18:55.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599613+0000", 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036176815000000001 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599646+0000", 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3192999999999999e-05 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599690+0000", 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4240000000000003e-05 2024-09-16T13:18:55.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64866 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3948544~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234856901000001, 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036355749999999999, 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64866 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563441+0000", 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563443+0000", 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0870000000000001e-06 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563448+0000", 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.062e-06 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563451+0000", 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599712+0000", 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036261007999999997 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599753+0000", 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0476999999999999e-05 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599796+0000", 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3226000000000003e-05 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64867 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3956736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234842161000003, 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036484356000000003, 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.833 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64867 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563456+0000", 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563458+0000", 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9980000000000002e-06 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563463+0000", 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563466+0000", 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599810+0000", 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036343683000000002 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599864+0000", 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3565e-05 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599939+0000", 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5889999999999993e-05 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64868 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3964928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234827002999999, 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036573319999999999, 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64868 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563471+0000", 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563473+0000", 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563479+0000", 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.178e-06 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563483+0000", 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6339999999999999e-06 2024-09-16T13:18:55.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599963+0000", 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036480090999999999 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600001+0000", 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8340999999999998e-05 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600044+0000", 2024-09-16T13:18:55.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64869 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3973120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234811028999999, 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036707818000000003, 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64869 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563487+0000", 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9100000000000003e-07 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563489+0000", 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563494+0000", 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563498+0000", 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.174e-06 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600061+0000", 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036563127000000001 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600119+0000", 2024-09-16T13:18:55.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8476000000000001e-05 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600194+0000", 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4869000000000004e-05 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64870 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3981312~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:55.846 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234795986999998, 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036852528000000002, 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64870 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563502+0000", 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563504+0000", 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9479999999999998e-06 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563509+0000", 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563512+0000", 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3239999999999999e-06 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600218+0000", 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036705812999999997 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600280+0000", 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2298000000000006e-05 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600354+0000", 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3343000000000006e-05 2024-09-16T13:18:55.850 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64871 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3989504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234781280999997, 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036938311000000001, 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.851 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64871 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563518+0000", 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9130000000000001e-06 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563523+0000", 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:18:55.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563529+0000", 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0710000000000002e-06 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036847958 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600411+0000", 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4388999999999999e-05 2024-09-16T13:18:55.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600454+0000", 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3411000000000002e-05 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64872 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3997696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234764773999999, 2024-09-16T13:18:55.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037077038999999999, 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64872 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563533+0000", 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563535+0000", 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1270000000000001e-06 2024-09-16T13:18:55.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563540+0000", 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3120000000000001e-06 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563544+0000", 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4699999999999998e-06 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600476+0000", 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036931691000000003 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600543+0000", 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7475e-05 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600610+0000", 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6397999999999997e-05 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.859 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64873 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4005888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234749749000002, 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037239282999999998, 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64873 2024-09-16T13:18:55.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563550+0000", 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563555+0000", 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563558+0000", 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2009999999999999e-06 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600630+0000", 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037071933000000001 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600702+0000", 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1245e-05 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600787+0000", 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5290000000000005e-05 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64874 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4014080~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234735303000001, 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037394846000000002, 2024-09-16T13:18:55.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64874 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0400000000000002e-07 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563565+0000", 2024-09-16T13:18:55.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1900000000000002e-06 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563570+0000", 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4199999999999998e-06 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563574+0000", 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600812+0000", 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037237600000000003 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4690000000000006e-05 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600957+0000", 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0463999999999994e-05 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.868 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64875 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4022272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234719507999998, 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537883000000001, 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64875 2024-09-16T13:18:55.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.65e-07 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563580+0000", 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563587+0000", 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2589999999999997e-06 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563591+0000", 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600982+0000", 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037390117 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601044+0000", 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2788999999999998e-05 2024-09-16T13:18:55.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601116+0000", 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1282e-05 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64876 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4030464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234702183000003, 2024-09-16T13:18:55.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037682571999999998, 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64876 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563596+0000", 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563613+0000", 2024-09-16T13:18:55.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7935999999999999e-05 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563620+0000", 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4740000000000002e-06 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563624+0000", 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601141+0000", 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037517170000000002 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601203+0000", 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2187999999999995e-05 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601278+0000", 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.877 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64877 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4038656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234668573999997, 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037813404000000002, 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64877 2024-09-16T13:18:55.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:18:55.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563631+0000", 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1330000000000002e-06 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563637+0000", 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563640+0000", 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601304+0000", 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037664438000000001 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601372+0000", 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7618000000000002e-05 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601442+0000", 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-05 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64878 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4046848~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234639401000003, 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037956456, 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.882 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64878 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563659+0000", 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563661+0000", 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-06 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563667+0000", 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3470000000000004e-06 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563671+0000", 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7170000000000002e-06 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601473+0000", 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037801955999999998 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601537+0000", 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3960000000000004e-05 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601614+0000", 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.886 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64879 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4055040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 42.234622762000001, 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038485017000000003, 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64879 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:18:55.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4099999999999998e-07 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563677+0000", 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1299999999999999e-06 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563682+0000", 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.889 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563688+0000", 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3329999999999996e-06 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601640+0000", 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037951864000000002 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601705+0000", 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5573000000000004e-05 2024-09-16T13:18:55.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.602160+0000", 2024-09-16T13:18:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000454411 2024-09-16T13:18:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:55.891 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:55.891 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:55.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:55.893 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:55.887+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:55.994 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:55.987+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:56.001 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:56.001 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:56.001 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:56.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:56.095 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:56.091+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:56.110 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:56.110 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:56.110 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:56.119 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:56.196 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:56.189+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:56.225 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:56.225 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 112.212923051, 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:18:56.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:18:56.229 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 112.21175612899999, 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:18:56.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 104.162412363, 2024-09-16T13:18:56.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.237 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:18:56.238 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 104.161802923, 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:18:56.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:18:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:56.243 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 103.664519109, 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.247 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 103.66341829300001, 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:18:56.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:18:56.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 103.66327865300001, 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:18:56.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 103.662778123, 2024-09-16T13:18:56.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:18:56.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:18:56.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 98.667608736999995, 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:18:56.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:18:56.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 88.411194895999998, 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.265 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.266 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:18:56.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 87.798631788999998, 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:56.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:18:56.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 60.960525619000002, 2024-09-16T13:18:56.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:18:56.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 60.596398286000003, 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:18:56.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:75420 3.4 e88/23 3:2605e3b8:::rbd_object_map.112d9326a5dd.0000000000000026:head v 88'165, mlcod=88'163)", 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.717669270000002, 2024-09-16T13:18:56.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055875079999999997, 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503433+0000", 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503451+0000", 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7640999999999999e-05 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503476+0000", 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5599999999999999e-05 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503495+0000", 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.825e-05 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503611+0000", 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011631799999999999 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503644+0000", 2024-09-16T13:18:56.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3476000000000003e-05 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.508954+0000", 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053093799999999998 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.509019+0000", 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5425999999999993e-05 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1195 3.4 e101/23 3:24bef2c8:::rbd_data.112da28a57db.0000000000000020:27 v 101'326, mlcod=99'322)", 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.847455644, 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005296243, 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373646+0000", 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7e-07 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373653+0000", 2024-09-16T13:18:56.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4000000000000003e-06 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374730+0000", 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00107654 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374741+0000", 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1474999999999999e-05 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374829+0000", 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7584999999999996e-05 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374854+0000", 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5269000000000002e-05 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378893+0000", 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040386770000000001 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378942+0000", 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1197 3.4 e101/23 3:26d31b55:::rbd_data.112da28a57db.000000000000000c:27 v 101'328, mlcod=99'322)", 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.846338571, 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046376150000000003, 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.292 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374763+0000", 2024-09-16T13:18:56.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5600000000000005e-07 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374784+0000", 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0781999999999999e-05 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374794+0000", 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0558e-05 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374800+0000", 2024-09-16T13:18:56.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375389+0000", 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058970500000000005 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375407+0000", 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7801000000000001e-05 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379351+0000", 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039435030000000001 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379400+0000", 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9332000000000001e-05 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.296 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:89341 3.4 e101/23 3:21ab973d:::rbd_data.112da28a57db.0000000000000002:head v 101'329, mlcod=101'326)", 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 18.723878778, 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005014859, 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497223+0000", 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8599999999999998e-07 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497231+0000", 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6709999999999999e-06 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497245+0000", 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.489e-05 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497253+0000", 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6199999999999999e-06 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498106+0000", 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00085257599999999996 2024-09-16T13:18:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498130+0000", 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4776000000000001e-05 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502154+0000", 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004023703 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502237+0000", 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2936999999999996e-05 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.4.0:2718 3.9 e106/23 3:9702159d:::rbd_data.112d404789fd.000000000000001c:22 v 106'440, mlcod=105'437)", 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.800060633999999, 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063154409999999998, 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421041+0000", 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:18:56.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421055+0000", 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4579999999999999e-05 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421080+0000", 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4859e-05 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421089+0000", 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8950000000000004e-06 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421166+0000", 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6892999999999997e-05 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421197+0000", 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1325000000000002e-05 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427304+0000", 2024-09-16T13:18:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061062449999999997 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427356+0000", 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2197000000000002e-05 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.4.0:2792 3.9 e110/23 3:938f1173:::rbd_data.112d94eb720f.0000000000000006:20 v 110'460, mlcod=106'457)", 2024-09-16T13:18:56.305 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.7654904659999993, 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041377590000000004, 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2300000000000001e-07 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455621+0000", 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0064000000000001e-05 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455658+0000", 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6288000000000002e-05 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455678+0000", 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0812999999999999e-05 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455817+0000", 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013879800000000001 2024-09-16T13:18:56.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455886+0000", 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8842000000000003e-05 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.459651+0000", 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037649519999999998 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.459748+0000", 2024-09-16T13:18:56.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7379000000000004e-05 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1263 3.4 e119/23 3:2111795e:::rbd_data.112d3e228c72.0000000000000003:18 v 119'349, mlcod=118'345)", 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:55.540473+0000", 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.68062763999999998, 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039574179999999999, 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540473+0000", 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540473+0000", 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540474+0000", 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540484+0000", 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0479000000000001e-05 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540494+0000", 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3910000000000004e-06 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540501+0000", 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7950000000000008e-06 2024-09-16T13:18:56.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540593+0000", 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1841000000000004e-05 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540622+0000", 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8597000000000001e-05 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:18:56.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.544386+0000", 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003763944 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.544431+0000", 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5039e-05 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.314 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:56.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:56.316 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:56.311+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:56.417 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:56.409+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:56.421 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:56.421 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:56.422 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:18:56.422 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:56.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:56.518 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:56.511+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:56.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:18:56.591 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:56.591 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:56.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:56.619 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:56.611+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:56.710 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:56.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62278 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390813465999997, 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027907498999999999, 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62278 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:18:56.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315829+0000", 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.79e-06 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315835+0000", 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.682e-06 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315838+0000", 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0730000000000001e-06 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343640+0000", 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027802384999999999 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343683+0000", 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3058e-05 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343734+0000", 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1104999999999999e-05 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390798904999997, 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:18:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390774939000003, 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:18:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:18:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:18:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390759768999999, 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:18:56.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390745346000003, 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:18:56.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:18:56.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.733 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390724101000004, 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:18:56.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:18:56.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.738 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390709532000002, 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:56.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:18:56.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390690763000002, 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:18:56.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390675868000002, 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:18:56.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390661657000003, 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.756 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390648288000001, 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:18:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:18:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390633606000002, 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:18:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:18:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390618437000001, 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:18:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390603796999997, 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:18:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390573068999998, 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:18:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:18:56.781 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390552051999997, 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.782 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:56.783 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:18:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390531944000003, 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.787 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.788 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.789 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:18:56.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390512041000001, 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.792 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.793 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:18:56.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390496828000003, 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:18:56.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.799 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 43.390481293999997, 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:56.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:18:56.803 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:56.804 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:56.804 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:56.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:56.806 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:56.799+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:56.907 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:56.901+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:56.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:18:56.914 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:56.914 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:56.914 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:56.914 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:56.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:57.008 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:57.001+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:57.026 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:57.027 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:57.027 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:57.027 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:57.027 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:57.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:57.044 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:57.066 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:57.086 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:57.109 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:57.103+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:57.112 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:57.137 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 43.923246435000003, 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:18:57.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:18:57.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.143 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75320 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1114112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.678006709000002, 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048391086999999999, 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75320 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455024+0000", 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455026+0000", 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455033+0000", 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8630000000000002e-06 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455037+0000", 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.608e-06 2024-09-16T13:18:57.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503178+0000", 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048141646000000003 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503284+0000", 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105917 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503415+0000", 2024-09-16T13:18:57.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130359 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75321 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1441792~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677989717000003, 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048644984000000002, 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.148 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75321 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455041+0000", 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455043+0000", 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0480000000000001e-06 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455048+0000", 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455052+0000", 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000002e-06 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503461+0000", 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048409079000000001 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503580+0000", 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119234 2024-09-16T13:18:57.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503685+0000", 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105006 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75322 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 2064384~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677964320000001, 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048835044000000001, 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.152 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75322 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2599999999999998e-07 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455068+0000", 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.092e-06 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455074+0000", 2024-09-16T13:18:57.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.925e-06 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455078+0000", 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503740+0000", 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048661514000000003 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503822+0000", 2024-09-16T13:18:57.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2196999999999999e-05 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503901+0000", 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9091000000000006e-05 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75344 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 589824~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677460680000003, 2024-09-16T13:18:57.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048412121000000002, 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75344 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455570+0000", 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455572+0000", 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2749999999999998e-06 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455578+0000", 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455582+0000", 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4789999999999998e-06 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503719+0000", 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048137054999999998 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:57.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503819+0000", 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9938000000000004e-05 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503982+0000", 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016225500000000001 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75345 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 917504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:57.160 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677444014000002, 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048702503000000001, 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75345 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455587+0000", 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455589+0000", 2024-09-16T13:18:57.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455594+0000", 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6840000000000001e-06 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455598+0000", 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7009999999999998e-06 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504027+0000", 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048429382999999999 2024-09-16T13:18:57.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504123+0000", 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5251e-05 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504289+0000", 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016599200000000001 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75346 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1212416~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677428657, 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048896186000000001, 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75346 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.165 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455602+0000", 2024-09-16T13:18:57.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2799999999999996e-07 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455604+0000", 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455609+0000", 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455615+0000", 2024-09-16T13:18:57.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2020000000000003e-06 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504315+0000", 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048700565000000001 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504382+0000", 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7088999999999994e-05 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504498+0000", 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115408 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75347 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1540096~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677410111, 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049121102999999999, 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.169 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75347 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7100000000000002e-07 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455623+0000", 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455629+0000", 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455633+0000", 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9400000000000004e-06 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504546+0000", 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048912718000000001 2024-09-16T13:18:57.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504623+0000", 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7726999999999996e-05 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504741+0000", 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117874 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75353 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 98304~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677274367000003, 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048929180000000003, 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75353 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:18:57.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455758+0000", 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8789999999999999e-06 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455763+0000", 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2190000000000003e-06 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455767+0000", 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504529+0000", 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048761566999999999 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504587+0000", 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8564999999999999e-05 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504685+0000", 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7754e-05 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75354 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 393216~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677239297, 2024-09-16T13:18:57.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049086520000000002, 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75354 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455794+0000", 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.508e-06 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455801+0000", 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5630000000000003e-06 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455805+0000", 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504704+0000", 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048898981000000001 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504759+0000", 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4330000000000003e-05 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504877+0000", 2024-09-16T13:18:57.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118608 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75355 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677220665, 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118086999999998, 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.183 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75355 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455810+0000", 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455812+0000", 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455818+0000", 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:18:57.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455821+0000", 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504914+0000", 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049092961999999997 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504928+0000", 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3210000000000001e-05 2024-09-16T13:18:57.186 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75356 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677204904, 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049129843999999999, 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.187 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75356 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455826+0000", 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0399999999999996e-07 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455828+0000", 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-06 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455835+0000", 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8410000000000002e-06 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455839+0000", 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2400000000000001e-06 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504947+0000", 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049108546000000003 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504955+0000", 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6109999999999996e-06 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75357 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.67718764, 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049141246, 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.191 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75357 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455843+0000", 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999998e-07 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455845+0000", 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999998e-06 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455852+0000", 2024-09-16T13:18:57.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455856+0000", 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6059999999999999e-06 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504974+0000", 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118385000000001 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504984+0000", 2024-09-16T13:18:57.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7880000000000001e-06 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75358 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677170691000001, 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049149627000000001, 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.195 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75358 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455860+0000", 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455862+0000", 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455867+0000", 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.395e-06 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455873+0000", 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0120000000000002e-06 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505002+0000", 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049128988999999998 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505009+0000", 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7950000000000002e-06 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.198 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75370 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677152937999999, 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049174830000000003, 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75370 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.199 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455878+0000", 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:18:57.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455880+0000", 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455885+0000", 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455889+0000", 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6090000000000001e-06 2024-09-16T13:18:57.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505040+0000", 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049151242999999997 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505052+0000", 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2055999999999999e-05 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75371 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677137801999997, 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049181864999999998, 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75371 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455893+0000", 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:18:57.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455895+0000", 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455900+0000", 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.536e-06 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455904+0000", 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505067+0000", 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049163224999999998 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505074+0000", 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.206 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75372 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677122664000002, 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049214870000000001, 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75372 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.207 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455908+0000", 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.89e-07 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455910+0000", 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455915+0000", 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455919+0000", 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.472e-06 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505109+0000", 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049190747 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505122+0000", 2024-09-16T13:18:57.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3009e-05 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75373 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677108146999998, 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049224933999999998, 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.211 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75373 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455924+0000", 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455930+0000", 2024-09-16T13:18:57.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4929999999999998e-06 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455936+0000", 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4459999999999998e-06 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505140+0000", 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049203510999999998 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505147+0000", 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1670000000000002e-06 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75374 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677090067999998, 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049235427999999998, 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.215 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75374 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:57.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455946+0000", 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860000000000001e-06 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455953+0000", 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5119999999999997e-06 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455956+0000", 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.579e-06 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505166+0000", 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049209527000000003 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505175+0000", 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75375 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.677069787999997, 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049243375999999998, 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:18:57.219 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75375 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:18:57.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455961+0000", 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455963+0000", 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729999999999999e-06 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455970+0000", 2024-09-16T13:18:57.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2520000000000002e-06 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455975+0000", 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505198+0000", 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049222852999999997 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:18:57.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505204+0000", 2024-09-16T13:18:57.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:18:57.223 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.223 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.223 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.223 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:18:57.223 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:18:57.223 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:57.224 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:57.224 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:57.225 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:57.225 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:57.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:18:57.226 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:57.227 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:18:57.227 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:57.217+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:57.325 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:57.319+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:57.375 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:57.376 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:57.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:18:57.426 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:57.421+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:57.480 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:57.480 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:57.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:18:57.527 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:57.525+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:18:57.598 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:57.598 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:57.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:18:57.628 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:57.623+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:57.697 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:18:57.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T13:18:57.703 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:57.703 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:57.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:18:57.729 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:57.721+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:57.830 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:57.825+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:57.859 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:57.860 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:57.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:18:57.931 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:57.923+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:57.971 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:57.971 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:57.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:18:58.032 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:58.025+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:58.077 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:58.077 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:58.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:18:58.133 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:18:58.127+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:18:58.186 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:58.186 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:58.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:18:58.235 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:58.233+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:58.288 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:58.288 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:58.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:18:58.335 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:58.327+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:58.407 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:58.407 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:58.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:18:58.435 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:58.429+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:58.518 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:58.519 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:58.527 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:18:58.536 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:58.529+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:58.626 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:58.626 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:58.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:18:58.639 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:58.633+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:58.737 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:58.737 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:58.739 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:58.733+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:58.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:18:58.840 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:18:58.835+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:18:58.842 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:58.842 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:58.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:18:58.941 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:18:58.933+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:18:58.951 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:58.951 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:58.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:18:58.972 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T13:18:58.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T13:18:59.041 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:59.033+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:59.061 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:59.061 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:59.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:18:59.142 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:18:59.137+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:18:59.212 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:59.213 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:59.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:18:59.227 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:18:59.243 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:59.235+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:59.310 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:59.310 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:59.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:18:59.345 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:18:59.339+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:18:59.395 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:18:59.412 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:18:59.412 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:18:59.413 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:18:59.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:18:59.430 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:18:59.445 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:18:59.437+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:18:59.449 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:18:59.469 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:18:59.491 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:18:59.512 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:18:59.514 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:18:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:18:59.514 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:18:59.514 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:59.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:18:59.533 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:18:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:18:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:18:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:18:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:18:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:18:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:18:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:18:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:18:59.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:18:59.590 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:59.585+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:59.647 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:59.641+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:59.656 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62278 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.336104616, 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027907498999999999, 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.657 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62278 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315829+0000", 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.79e-06 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315835+0000", 2024-09-16T13:18:59.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.682e-06 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315838+0000", 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0730000000000001e-06 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343640+0000", 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027802384999999999 2024-09-16T13:18:59.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343683+0000", 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3058e-05 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343734+0000", 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1104999999999999e-05 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.336090055, 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.662 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:18:59.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.336066088999999, 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.336050919000002, 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.671 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:18:59.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.336036495999998, 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.675 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:59.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.679 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.336015250999999, 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:18:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.336000681999998, 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.684 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:18:59.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.688 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335981912999998, 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:18:59.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:18:59.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335967017999998, 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.693 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:18:59.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.697 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335952806999998, 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:18:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:18:59.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335939437999997, 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:18:59.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:18:59.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:59.706 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335924755999997, 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:18:59.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335909587000003, 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:59.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:18:59.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335894947, 2024-09-16T13:18:59.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:18:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335864219000001, 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.720 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:18:59.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335843202, 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.724 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.728 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335823093999998, 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.729 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:18:59.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:18:59.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:59.733 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335803190999997, 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:18:59.737 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335787977999999, 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.738 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:18:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:18:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:59.742 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.335772444, 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:18:59.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:18:59.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:18:59.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.747 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:18:59.747 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:18:59.747 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:18:59.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:18:59.749 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:59.743+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:59.849 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:18:59.843+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:18:59.857 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:18:59.858 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:59.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:18:59.950 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:18:59.941+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:18:59.967 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:18:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:18:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:18:59.967 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:18:59.967 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:18:59.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:18:59.982 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T13:18:59.983 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:18:59.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T13:19:00.051 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:00.043+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:00.078 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:00.078 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:00.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:00.078 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:00.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.864021911999998, 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:19:00.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75320 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1114112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.618782185999997, 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048391086999999999, 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.084 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75320 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455024+0000", 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455026+0000", 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455033+0000", 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8630000000000002e-06 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455037+0000", 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.608e-06 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503178+0000", 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048141646000000003 2024-09-16T13:19:00.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503284+0000", 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105917 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503415+0000", 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130359 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75321 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1441792~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.618765193999998, 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048644984000000002, 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75321 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455041+0000", 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455043+0000", 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0480000000000001e-06 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455048+0000", 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455052+0000", 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000002e-06 2024-09-16T13:19:00.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503461+0000", 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048409079000000001 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503580+0000", 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119234 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503685+0000", 2024-09-16T13:19:00.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105006 2024-09-16T13:19:00.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.093 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.093 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75322 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 2064384~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.093 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.618739797000003, 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048835044000000001, 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75322 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2599999999999998e-07 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455068+0000", 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.092e-06 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455074+0000", 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.925e-06 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455078+0000", 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:19:00.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503740+0000", 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048661514000000003 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503822+0000", 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2196999999999999e-05 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503901+0000", 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9091000000000006e-05 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75344 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 589824~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.618236156999998, 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048412121000000002, 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75344 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455570+0000", 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455572+0000", 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2749999999999998e-06 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455578+0000", 2024-09-16T13:19:00.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455582+0000", 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4789999999999998e-06 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503719+0000", 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048137054999999998 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503819+0000", 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9938000000000004e-05 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503982+0000", 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016225500000000001 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75345 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 917504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.102 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.618219490999998, 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048702503000000001, 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75345 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:19:00.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455587+0000", 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455589+0000", 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455594+0000", 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6840000000000001e-06 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455598+0000", 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7009999999999998e-06 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504027+0000", 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048429382999999999 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504123+0000", 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5251e-05 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504289+0000", 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016599200000000001 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.106 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75346 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1212416~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.618204134000003, 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048896186000000001, 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75346 2024-09-16T13:19:00.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455602+0000", 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2799999999999996e-07 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455604+0000", 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455609+0000", 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455615+0000", 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2020000000000003e-06 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504315+0000", 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048700565000000001 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504382+0000", 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7088999999999994e-05 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504498+0000", 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115408 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75347 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1540096~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.618185588000003, 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049121102999999999, 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.111 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75347 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7100000000000002e-07 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455623+0000", 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455629+0000", 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455633+0000", 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9400000000000004e-06 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504546+0000", 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048912718000000001 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504623+0000", 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7726999999999996e-05 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504741+0000", 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117874 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.115 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75353 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 98304~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.618049843999998, 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048929180000000003, 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75353 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455758+0000", 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8789999999999999e-06 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455763+0000", 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2190000000000003e-06 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455767+0000", 2024-09-16T13:19:00.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504529+0000", 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048761566999999999 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504587+0000", 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8564999999999999e-05 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504685+0000", 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7754e-05 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75354 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 393216~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.618014774000002, 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049086520000000002, 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.120 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75354 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455794+0000", 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.508e-06 2024-09-16T13:19:00.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455801+0000", 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5630000000000003e-06 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455805+0000", 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504704+0000", 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048898981000000001 2024-09-16T13:19:00.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504759+0000", 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4330000000000003e-05 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504877+0000", 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118608 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75355 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.617996142000003, 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118086999999998, 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75355 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455810+0000", 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455812+0000", 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455818+0000", 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455821+0000", 2024-09-16T13:19:00.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504914+0000", 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049092961999999997 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504928+0000", 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3210000000000001e-05 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75356 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.617980381000002, 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049129843999999999, 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75356 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455826+0000", 2024-09-16T13:19:00.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0399999999999996e-07 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455828+0000", 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-06 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455835+0000", 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8410000000000002e-06 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455839+0000", 2024-09-16T13:19:00.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2400000000000001e-06 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504947+0000", 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049108546000000003 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504955+0000", 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6109999999999996e-06 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.132 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75357 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.617963117000002, 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049141246, 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75357 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.133 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455843+0000", 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999998e-07 2024-09-16T13:19:00.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455845+0000", 2024-09-16T13:19:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999998e-06 2024-09-16T13:19:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455852+0000", 2024-09-16T13:19:00.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455856+0000", 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6059999999999999e-06 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504974+0000", 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118385000000001 2024-09-16T13:19:00.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504984+0000", 2024-09-16T13:19:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7880000000000001e-06 2024-09-16T13:19:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.137 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75358 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.617946168000003, 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049149627000000001, 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75358 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455860+0000", 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455862+0000", 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455867+0000", 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.395e-06 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455873+0000", 2024-09-16T13:19:00.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0120000000000002e-06 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505002+0000", 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049128988999999998 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505009+0000", 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7950000000000002e-06 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75370 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.617928415000002, 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049174830000000003, 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75370 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.142 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455878+0000", 2024-09-16T13:19:00.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455880+0000", 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455885+0000", 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455889+0000", 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6090000000000001e-06 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505040+0000", 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049151242999999997 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505052+0000", 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2055999999999999e-05 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.145 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75371 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.617913279, 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049181864999999998, 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75371 2024-09-16T13:19:00.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455893+0000", 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455895+0000", 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455900+0000", 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.536e-06 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455904+0000", 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505067+0000", 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049163224999999998 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505074+0000", 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:19:00.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75372 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.617898140999998, 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049214870000000001, 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.150 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75372 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455908+0000", 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.89e-07 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455910+0000", 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455915+0000", 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455919+0000", 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.472e-06 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505109+0000", 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049190747 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505122+0000", 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3009e-05 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75373 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.617883624000001, 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049224933999999998, 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.154 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75373 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:00.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455924+0000", 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455930+0000", 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4929999999999998e-06 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455936+0000", 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4459999999999998e-06 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505140+0000", 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049203510999999998 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505147+0000", 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1670000000000002e-06 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75374 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.617865545000001, 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049235427999999998, 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75374 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455946+0000", 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860000000000001e-06 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455953+0000", 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5119999999999997e-06 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455956+0000", 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.579e-06 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505166+0000", 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049209527000000003 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505175+0000", 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75375 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 40.617845265, 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049243375999999998, 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75375 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455961+0000", 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455963+0000", 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729999999999999e-06 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455970+0000", 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2520000000000002e-06 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455975+0000", 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505198+0000", 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049222852999999997 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505204+0000", 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.165 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:00.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:19:00.167 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:00.159+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:00.267 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:00.259+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:00.316 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:00.316 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:00.316 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:00.316 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:00.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:19:00.369 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:00.363+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:00.428 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:00.429 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:00.429 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:00.429 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:00.429 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:00.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:19:00.480 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:00.475+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:00.570 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:00.565+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 59.760626232999996, 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.588 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:19:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:19:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:19:00.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82800 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1130496~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724568138000002, 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050353660000000001, 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82800 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858530+0000", 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858534+0000", 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8229999999999997e-06 2024-09-16T13:19:00.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858547+0000", 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3302000000000001e-05 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858557+0000", 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-05 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908637+0000", 2024-09-16T13:19:00.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050079811000000002 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908731+0000", 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4085000000000005e-05 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908883+0000", 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000151676 2024-09-16T13:19:00.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82801 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1212416~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724532242999999, 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050667642999999998, 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82801 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:19:00.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858566+0000", 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3499999999999995e-07 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858570+0000", 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8299999999999998e-06 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858590+0000", 2024-09-16T13:19:00.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9831e-05 2024-09-16T13:19:00.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858597+0000", 2024-09-16T13:19:00.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6729999999999992e-06 2024-09-16T13:19:00.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908930+0000", 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050332492999999999 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909036+0000", 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010603999999999999 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909233+0000", 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019704100000000001 2024-09-16T13:19:00.604 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82802 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724491946999997, 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050929965000000001, 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.605 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82802 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:00.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:19:00.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858610+0000", 2024-09-16T13:19:00.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0820000000000001e-06 2024-09-16T13:19:00.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858655+0000", 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4901000000000003e-05 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858668+0000", 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2343e-05 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909278+0000", 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050610005 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909374+0000", 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6399999999999999e-05 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909535+0000", 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016137599999999999 2024-09-16T13:19:00.609 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82803 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1474560~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:19:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724383095999997, 2024-09-16T13:19:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051051886999999997, 2024-09-16T13:19:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.610 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82803 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858715+0000", 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999995e-07 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858719+0000", 2024-09-16T13:19:00.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.732e-06 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858728+0000", 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6730000000000004e-06 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858733+0000", 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.532e-06 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909572+0000", 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050839399 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909647+0000", 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5073000000000004e-05 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909766+0000", 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118805 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.614 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82804 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1654784~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724358723999998, 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051320085000000001, 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82804 2024-09-16T13:19:00.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3399999999999998e-07 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858742+0000", 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4150000000000002e-06 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858748+0000", 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0090000000000004e-06 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858753+0000", 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1649999999999997e-06 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909812+0000", 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051058612000000003 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909907+0000", 2024-09-16T13:19:00.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5795e-05 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910059+0000", 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015135499999999999 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82805 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1736704~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:00.619 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724338807999999, 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051834002999999997, 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82805 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:00.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858762+0000", 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4959999999999999e-06 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858768+0000", 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8830000000000003e-06 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858771+0000", 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9319999999999998e-06 2024-09-16T13:19:00.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910078+0000", 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051306696999999998 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910163+0000", 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5065e-05 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910593+0000", 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042940799999999999 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82806 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1916928~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724322026999999, 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052120149999999997, 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.624 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82806 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:19:00.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858776+0000", 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2600000000000002e-07 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858778+0000", 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3070000000000001e-06 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858784+0000", 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858788+0000", 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8870000000000002e-06 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910643+0000", 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051854355999999997 2024-09-16T13:19:00.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910735+0000", 2024-09-16T13:19:00.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2454000000000006e-05 2024-09-16T13:19:00.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910896+0000", 2024-09-16T13:19:00.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016055600000000001 2024-09-16T13:19:00.628 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82807 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1998848~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724302858999998, 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052345503000000002, 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.629 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82807 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858797+0000", 2024-09-16T13:19:00.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858804+0000", 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0079999999999996e-06 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858808+0000", 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910935+0000", 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052126695000000001 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911010+0000", 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4764000000000004e-05 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911140+0000", 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013022500000000001 2024-09-16T13:19:00.633 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82830 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 770048~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724284849, 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052616234999999997, 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.634 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82830 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858815+0000", 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3130000000000001e-06 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858821+0000", 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7200000000000003e-06 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858838+0000", 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6492000000000001e-05 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911178+0000", 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052340684999999998 2024-09-16T13:19:00.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911272+0000", 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4106999999999999e-05 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911429+0000", 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156389 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82831 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724251062, 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052803142999999997, 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82831 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.639 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858848+0000", 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.209e-06 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858858+0000", 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0111e-05 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858870+0000", 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1976e-05 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858876+0000", 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5649999999999998e-06 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911449+0000", 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052573110999999999 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911521+0000", 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1209000000000002e-05 2024-09-16T13:19:00.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911650+0000", 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012896199999999999 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82832 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1032192~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724213401, 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053031663999999999, 2024-09-16T13:19:00.643 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82832 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858885+0000", 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6899999999999996e-07 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858889+0000", 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.087e-06 2024-09-16T13:19:00.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858905+0000", 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5744999999999999e-05 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858909+0000", 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7269999999999998e-06 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911673+0000", 2024-09-16T13:19:00.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052763697999999998 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911767+0000", 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4054000000000004e-05 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911916+0000", 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148584 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.647 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82833 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1114112~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724180298, 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053225354000000002, 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82833 2024-09-16T13:19:00.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858918+0000", 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858922+0000", 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.371e-06 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858933+0000", 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1076e-05 2024-09-16T13:19:00.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858940+0000", 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9840000000000004e-06 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911944+0000", 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053004044 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912021+0000", 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7189999999999998e-05 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912143+0000", 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012160200000000001 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82834 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1294336~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.652 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724150545999997, 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053469475000000002, 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82834 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858948+0000", 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858952+0000", 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858970+0000", 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.889e-05 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858978+0000", 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912172+0000", 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053193670999999998 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912268+0000", 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6100000000000005e-05 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912416+0000", 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014817500000000001 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82835 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1376256~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724110252000003, 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053671286999999998, 2024-09-16T13:19:00.657 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82835 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858988+0000", 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858992+0000", 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858999+0000", 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859003+0000", 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912456+0000", 2024-09-16T13:19:00.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053453374999999997 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912526+0000", 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9237000000000001e-05 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912658+0000", 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013284000000000001 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82836 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1556480~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724090404999998, 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053937880000000001, 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82836 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859008+0000", 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859010+0000", 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1950000000000002e-06 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859016+0000", 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8429999999999998e-06 2024-09-16T13:19:00.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859019+0000", 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912697+0000", 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053677322 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912796+0000", 2024-09-16T13:19:00.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8991000000000001e-05 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912945+0000", 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149207 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82837 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1638400~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724074381999998, 2024-09-16T13:19:00.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054154068999999999, 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82837 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859024+0000", 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859026+0000", 2024-09-16T13:19:00.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859031+0000", 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6690000000000003e-06 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859035+0000", 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6239999999999999e-06 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912981+0000", 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053946409000000001 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913054+0000", 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3052999999999993e-05 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913177+0000", 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122803 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.670 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82838 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1818624~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724058943999999, 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054416554999999998, 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82838 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859041+0000", 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2139999999999999e-06 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859047+0000", 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6359999999999999e-06 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859051+0000", 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6519999999999999e-06 2024-09-16T13:19:00.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913215+0000", 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054164344000000003 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913307+0000", 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2546999999999993e-05 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913455+0000", 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147639 2024-09-16T13:19:00.674 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.675 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.675 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.675 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82839 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1900544~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.675 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:19:00.675 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724043281999997, 2024-09-16T13:19:00.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054634963000000002, 2024-09-16T13:19:00.675 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.675 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82839 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859055+0000", 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859057+0000", 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3910000000000001e-06 2024-09-16T13:19:00.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859066+0000", 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5150000000000006e-06 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859072+0000", 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5810000000000002e-06 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913493+0000", 2024-09-16T13:19:00.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054420751000000003 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913563+0000", 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9938999999999996e-05 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913689+0000", 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012626299999999999 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.681 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82840 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 2080768~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.724016245999998, 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054884993999999999, 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82840 2024-09-16T13:19:00.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:00.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859082+0000", 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5799999999999998e-07 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859086+0000", 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7510000000000002e-06 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859097+0000", 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1085e-05 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859104+0000", 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-06 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913718+0000", 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054614085 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913814+0000", 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5514999999999996e-05 2024-09-16T13:19:00.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:00.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:00.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:00.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913966+0000", 2024-09-16T13:19:00.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015253 2024-09-16T13:19:00.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.686 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:00.686 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:00.686 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:00.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:19:00.688 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:00.679+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:00.789 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:00.783+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:00.793 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:00.794 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:00.794 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:00.794 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:00.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:19:00.890 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:00.885+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:00.905 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:00.905 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:00.905 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:00.905 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:00.906 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:00.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:19:00.991 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:00.983+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:00.996 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T13:19:01.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T13:19:01.023 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:01.023 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:01.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:01.023 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:01.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.023 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.023 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137436020999999, 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:19:01.027 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137417186, 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.028 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:01.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.032 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137398046999998, 2024-09-16T13:19:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:19:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:19:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.033 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:19:01.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137380989, 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:19:01.037 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:01.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:19:01.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:19:01.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:19:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:19:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.042 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137363628999999, 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.043 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:19:01.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137345920000001, 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:19:01.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137328621000002, 2024-09-16T13:19:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:19:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.052 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:19:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.053 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:19:01.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137293522999997, 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.058 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.062 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137270368000003, 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:01.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.066 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137229349000002, 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.067 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:19:01.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.071 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137206448999997, 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:19:01.075 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137181400000003, 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.076 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:19:01.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137156179000002, 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:19:01.080 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:19:01.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.137137813999999, 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:19:01.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.089 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.136874677000002, 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:19:01.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.093 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.136849669, 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:19:01.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:19:01.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:19:01.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.136821337000001, 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:19:01.098 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:19:01.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:19:01.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.136532639000002, 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:19:01.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.136512529000001, 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.107 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.111 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.132311604999998, 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.115 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.116 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.116 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:01.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:19:01.118 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:01.111+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:01.218 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:01.211+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:01.265 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:01.265 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:01.265 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:01.265 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:01.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:19:01.320 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:01.315+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:01.374 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:01.375 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:01.375 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:01.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:19:01.420 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:01.413+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:01.488 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90616 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:head [delete] snapc 21=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.914091561999999, 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.870938027, 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90616 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570781+0000", 2024-09-16T13:19:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.88e-06 2024-09-16T13:19:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570792+0000", 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0567000000000001e-05 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570816+0000", 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-05 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570850+0000", 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3779999999999998e-05 2024-09-16T13:19:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570946+0000", 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6294999999999999e-05 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570958+0000", 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438058+0000", 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86709994199999996 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438302+0000", 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024435100000000002 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438940+0000", 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063796000000000002 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441545+0000", 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026055509999999998 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441576+0000", 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0247e-05 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441622+0000", 2024-09-16T13:19:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6047999999999997e-05 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441716+0000", 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4524000000000005e-05 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90621 3.e 3.2a047d2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.913990322, 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.860052445, 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90621 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570881+0000", 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:19:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570885+0000", 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7230000000000004e-06 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570901+0000", 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5756999999999998e-05 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570908+0000", 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2960000000000001e-06 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571040+0000", 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131803 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571069+0000", 2024-09-16T13:19:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8694e-05 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430917+0000", 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85984788899999998 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430932+0000", 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5129e-05 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90624 3.e 3.2071842e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.913407058000001, 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85950820299999997, 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90624 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571465+0000", 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1799999999999999e-06 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571490+0000", 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5320000000000002e-05 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571524+0000", 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4190000000000003e-05 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571543+0000", 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8674000000000001e-05 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571647+0000", 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103803 2024-09-16T13:19:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571669+0000", 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2354e-05 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430962+0000", 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85929225799999998 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430971+0000", 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4239999999999999e-06 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2775 3.1b e108/107 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:23 v 108'605, mlcod=108'601)", 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.922696822999999, 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069960880000000001, 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562174+0000", 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7000000000000004e-07 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562181+0000", 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2459999999999997e-06 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562194+0000", 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2945000000000001e-05 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562200+0000", 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3679999999999998e-06 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562237+0000", 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7039999999999998e-05 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562262+0000", 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4443999999999998e-05 2024-09-16T13:19:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569096+0000", 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068345039999999999 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569169+0000", 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2770999999999998e-05 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(3.12 109/108 [PushOp(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22, version: 83'227, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22@83'227, size: 1048576, copy_subset: [0~1048576], clone_subset: {}, snapset: 22=[]:{22=[22]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.824561159, 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056147790000000003, 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660310+0000", 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-06 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662684+0000", 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023738169999999999 2024-09-16T13:19:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662716+0000", 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2271999999999999e-05 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662753+0000", 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6851999999999999e-05 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662882+0000", 2024-09-16T13:19:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012939700000000001 2024-09-16T13:19:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662892+0000", 2024-09-16T13:19:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0315e-05 2024-09-16T13:19:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.665923+0000", 2024-09-16T13:19:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030311140000000001 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90825 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch reconnect cookie 140083147222288 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.033856108, 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0248692960000001, 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90825 2024-09-16T13:19:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451018+0000", 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8569999999999997e-06 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451032+0000", 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4586999999999999e-05 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451078+0000", 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6180000000000002e-05 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451101+0000", 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2773e-05 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451261+0000", 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016003600000000001 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451276+0000", 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4387e-05 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459307+0000", 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080310969999999992 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459318+0000", 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1136e-05 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471312+0000", 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.011993798 2024-09-16T13:19:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471508+0000", 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019619700000000001 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.475883+0000", 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004375248 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2793 3.1b e110/107 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 v 110'613, mlcod=108'609)", 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.028693286999999, 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070642279999999997, 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456178+0000", 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.517e-06 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456230+0000", 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2323e-05 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456251+0000", 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0619999999999999e-05 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456269+0000", 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7883000000000001e-05 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456632+0000", 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036355300000000002 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456686+0000", 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3454e-05 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463210+0000", 2024-09-16T13:19:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065238550000000003 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463241+0000", 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1022999999999998e-05 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90826 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.028629029999999, 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023049584, 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90826 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456244+0000", 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3000000000000002e-06 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456261+0000", 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7036000000000001e-05 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456294+0000", 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456313+0000", 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8876999999999999e-05 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456444+0000", 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013131300000000001 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456462+0000", 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7920000000000001e-05 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459343+0000", 2024-09-16T13:19:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028811539999999999 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459351+0000", 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5870000000000004e-06 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472631+0000", 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0132794970000001 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472737+0000", 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106496 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.473418+0000", 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068136 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.474875+0000", 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001456564 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479175+0000", 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042996450000000004 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479238+0000", 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3610999999999998e-05 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479290+0000", 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2163999999999997e-05 2024-09-16T13:19:01.535 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1617 3.12 e111/110 3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22 v 111'330, mlcod=0'0)", 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.018899208000001, 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060955660000000002, 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465972+0000", 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2999999999999999e-07 2024-09-16T13:19:01.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465984+0000", 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2133999999999999e-05 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465998+0000", 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4184e-05 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466009+0000", 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1371e-05 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467605+0000", 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015953129999999999 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467632+0000", 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7447e-05 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472017+0000", 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043852589999999999 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472066+0000", 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8928000000000003e-05 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.540 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1618 3.12 e111/110 3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22 v 111'332, mlcod=0'0)", 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.018302552, 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00616418, 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466568+0000", 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6000000000000003e-07 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466575+0000", 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8809999999999998e-06 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466899+0000", 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032458399999999999 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466912+0000", 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2767e-05 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469208+0000", 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022958369999999998 2024-09-16T13:19:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469231+0000", 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2578e-05 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472690+0000", 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003459443 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472731+0000", 2024-09-16T13:19:01.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1430000000000001e-05 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1621 3.12 e111/110 3:4a3469d0:::rbd_data.112d12abb20f.000000000000003a:28 v 111'338, mlcod=111'334)", 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.996394991000001, 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005747411, 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488476+0000", 2024-09-16T13:19:01.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488497+0000", 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1542000000000001e-05 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488508+0000", 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0461e-05 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488514+0000", 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2990000000000002e-06 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488568+0000", 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4559999999999999e-05 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488591+0000", 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2416e-05 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494170+0000", 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055793279999999997 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494222+0000", 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2079000000000001e-05 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90830 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.995794348, 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054188910000000003, 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90830 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489077+0000", 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.544e-06 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489084+0000", 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5919999999999997e-06 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489111+0000", 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7484999999999999e-05 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489140+0000", 2024-09-16T13:19:01.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8466000000000002e-05 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489192+0000", 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1946e-05 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489255+0000", 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3751000000000007e-05 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:01.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489369+0000", 2024-09-16T13:19:01.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011379000000000001 2024-09-16T13:19:01.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:01.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494232+0000", 2024-09-16T13:19:01.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048629679999999996 2024-09-16T13:19:01.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494426+0000", 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000194258 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494467+0000", 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0339999999999997e-05 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494494+0000", 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7750999999999999e-05 2024-09-16T13:19:01.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1622 3.12 e111/110 3:498ca772:::rbd_data.112d12abb20f.0000000000000017:28 v 111'340, mlcod=111'334)", 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.995375336, 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074768710000000004, 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:19:01.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489503+0000", 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8720000000000002e-06 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489515+0000", 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2191000000000001e-05 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489524+0000", 2024-09-16T13:19:01.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6170000000000007e-06 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489592+0000", 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7670999999999996e-05 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489614+0000", 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2112000000000001e-05 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496931+0000", 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073173739999999998 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496971+0000", 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0247999999999998e-05 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90832 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch unwatch cookie 140083147222288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:19:01.560 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.990082593, 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055530110000000001, 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90832 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494788+0000", 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3900000000000004e-07 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494791+0000", 2024-09-16T13:19:01.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9050000000000001e-06 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494799+0000", 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3100000000000001e-06 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494804+0000", 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2170000000000002e-06 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496030+0000", 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012255689999999999 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496148+0000", 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117612 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496408+0000", 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026060999999999999 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.499840+0000", 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034319670000000002 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500173+0000", 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033327399999999999 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500232+0000", 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8303000000000001e-05 2024-09-16T13:19:01.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500340+0000", 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010860499999999999 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:91826 3.11 3:8c261a44:::rbd_data.112d770f4ae2.000000000000000e:head [delete] snapc 1c=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.8382984489999998, 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077560520000000003, 2024-09-16T13:19:01.566 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 91826 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:01.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646576+0000", 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2699999999999998e-06 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646592+0000", 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6439999999999998e-05 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646637+0000", 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5120000000000002e-05 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646657+0000", 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9737000000000001e-05 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646782+0000", 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012466699999999999 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646790+0000", 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6680000000000007e-06 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651149+0000", 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043590959999999998 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651538+0000", 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000389071 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651714+0000", 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017587200000000001 2024-09-16T13:19:01.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654242+0000", 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025272659999999998 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654249+0000", 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.61e-06 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654294+0000", 2024-09-16T13:19:01.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4644000000000001e-05 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654327+0000", 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3590999999999997e-05 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92599 3.11 3:8b1d31ff:::rbd_data.112d3e228c72.0000000000000004:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8454980179999998, 2024-09-16T13:19:01.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070407359999999997, 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92599 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639375+0000", 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9399999999999998e-06 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639386+0000", 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1606999999999999e-05 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639426+0000", 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9329000000000002e-05 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639439+0000", 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.348e-05 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639642+0000", 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000202976 2024-09-16T13:19:01.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639865+0000", 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022277500000000001 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640355+0000", 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049025000000000002 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:01.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.644293+0000", 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003937739 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646196+0000", 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019030410000000001 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646332+0000", 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013587899999999999 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646413+0000", 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.072e-05 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92601 3.13 3:cbdfcc30:::rbd_data.112d3e228c72.0000000000000006:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8453963780000002, 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069350899999999997, 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.579 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92601 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639475+0000", 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.59e-06 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639482+0000", 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.46e-06 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639568+0000", 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6064999999999997e-05 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639591+0000", 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3564e-05 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639945+0000", 2024-09-16T13:19:01.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035337399999999999 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640456+0000", 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051193699999999996 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640736+0000", 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027973199999999998 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.643326+0000", 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025895129999999999 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646146+0000", 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028207359999999999 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646277+0000", 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013046999999999999 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646409+0000", 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013164900000000001 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92605 3.11 3:8caec98f:::rbd_data.112d3e228c72.000000000000000a:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.8429616070000003, 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067373449999999996, 2024-09-16T13:19:01.585 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92605 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641911+0000", 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-06 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641925+0000", 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.403e-05 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641959+0000", 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3448999999999998e-05 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641976+0000", 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7397e-05 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.642102+0000", 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012618899999999999 2024-09-16T13:19:01.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.642511+0000", 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040815799999999999 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.643089+0000", 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057817299999999997 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.644631+0000", 2024-09-16T13:19:01.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015421969999999999 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648481+0000", 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038498930000000001 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648543+0000", 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2477000000000005e-05 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648646+0000", 2024-09-16T13:19:01.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010228199999999999 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:93417 3.e 3:74f3d54d:::rbd_data.112d30ff5030.000000000000000b:head [delete] snapc 13=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.3877575659999999, 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077949680000000002, 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 93417 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097116+0000", 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3859999999999999e-06 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097141+0000", 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5732e-05 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097191+0000", 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9854000000000001e-05 2024-09-16T13:19:01.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097211+0000", 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0046000000000001e-05 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097304+0000", 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2318999999999995e-05 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097311+0000", 2024-09-16T13:19:01.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4089999999999997e-06 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.101746+0000", 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044344780000000004 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.102087+0000", 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034153299999999998 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:01.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.102361+0000", 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027436799999999999 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104058+0000", 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001696431 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104805+0000", 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0007473 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104848+0000", 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3112999999999999e-05 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104907+0000", 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8999000000000002e-05 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.597 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:93579 3.11 3:8cf36ad7:::rbd_data.112d8b6bb15.0000000000000010:head [delete] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:01.130561+0000", 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.35430867500000002, 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0053757259999999999, 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 93579 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:01.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130561+0000", 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130561+0000", 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130563+0000", 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5e-06 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130570+0000", 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4800000000000004e-06 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130590+0000", 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0023000000000001e-05 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130603+0000", 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2656e-05 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130752+0000", 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014950799999999999 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.131228+0000", 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00047526099999999999 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.131487+0000", 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025928099999999999 2024-09-16T13:19:01.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.134000+0000", 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00251294 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.135812+0000", 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018124739999999999 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.135888+0000", 2024-09-16T13:19:01.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6032000000000007e-05 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.135937+0000", 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8571e-05 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:01.603 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:01.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:19:01.605 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:01.597+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:01.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:19:01.705 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:01.699+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:01.746 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:19:01.753 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:01.753 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:01.763 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:19:01.767 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:19:01.786 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:19:01.805 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:19:01.806 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:01.803+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:01.826 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:19:01.847 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:19:01.861 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:01.861 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:01.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:19:01.870 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:19:01.893 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:19:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:19:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:19:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:19:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:19:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:19:01.895 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:19:01.896 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:19:01.896 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:19:01.907 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:01.899+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:01.965 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:01.965 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:01.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:19:01.996 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:19:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T13:19:02.007 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:01.999+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:02.009 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T13:19:02.082 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:02.083 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:02.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:19:02.109 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:02.103+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:02.205 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:02.209 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:02.203+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:02.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:19:02.310 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:02.303+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:02.317 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:02.317 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:02.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:19:02.411 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:02.405+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:02.416 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:02.416 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:02.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:19:02.513 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:02.505+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:02.516 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:02.516 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:02.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:19:02.613 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:02.605+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:02.675 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:02.675 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:02.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:19:02.716 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:02.711+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:02.785 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:02.785 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:02.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:19:02.815 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:02.811+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:02.895 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:02.895 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:02.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:19:02.916 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:02.907+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:19:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T13:19:02.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T13:19:03.015 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:03.015 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:03.018 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:03.013+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:03.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:19:03.118 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:03.109+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:03.139 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:03.139 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:03.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:19:03.219 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:03.211+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:03.247 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:03.247 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:03.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:19:03.319 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:03.311+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:03.353 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:03.353 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:03.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:19:03.420 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:03.417+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:03.507 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:03.508 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:03.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:19:03.520 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:03.517+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:03.619 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:03.619 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:03.621 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:03.615+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:03.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:19:03.722 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:03.717+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:03.726 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:03.726 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:03.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:19:03.823 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:03.817+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:03.836 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:03.837 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:03.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:19:03.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:19:03.941 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:03.942 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:03.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:19:03.954 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:03.949+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:04.011 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T13:19:04.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T13:19:04.025 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:04.019+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:04.044 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:19:04.055 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:04.055 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:04.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:19:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:19:04.084 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:19:04.110 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:19:04.125 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:04.119+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:04.136 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:19:04.161 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:19:04.181 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:19:04.214 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:19:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:19:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:19:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:19:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:19:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:19:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:19:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:19:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:19:04.219 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:04.219 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:04.219 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:04.219 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:04.226 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:04.219+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:04.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:19:04.327 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:04.321+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:04.372 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:04.372 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:04.373 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:04.373 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:04.373 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:04.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:19:04.428 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:04.421+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:04.483 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90616 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:head [delete] snapc 21=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 20.907690509999998, 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.870938027, 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90616 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570781+0000", 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.88e-06 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570792+0000", 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0567000000000001e-05 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570816+0000", 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-05 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570850+0000", 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3779999999999998e-05 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570946+0000", 2024-09-16T13:19:04.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6294999999999999e-05 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570958+0000", 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438058+0000", 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86709994199999996 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438302+0000", 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024435100000000002 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438940+0000", 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063796000000000002 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441545+0000", 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026055509999999998 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441576+0000", 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0247e-05 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441622+0000", 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6047999999999997e-05 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441716+0000", 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4524000000000005e-05 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.490 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90621 3.e 3.2a047d2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 20.907589269999999, 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.860052445, 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.491 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90621 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570881+0000", 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570885+0000", 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7230000000000004e-06 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570901+0000", 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5756999999999998e-05 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570908+0000", 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2960000000000001e-06 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571040+0000", 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131803 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571069+0000", 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8694e-05 2024-09-16T13:19:04.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430917+0000", 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85984788899999998 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430932+0000", 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5129e-05 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90624 3.e 3.2071842e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 20.907006006, 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85950820299999997, 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90624 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571465+0000", 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1799999999999999e-06 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571490+0000", 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5320000000000002e-05 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571524+0000", 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4190000000000003e-05 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571543+0000", 2024-09-16T13:19:04.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8674000000000001e-05 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571647+0000", 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103803 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571669+0000", 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2354e-05 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430962+0000", 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85929225799999998 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430971+0000", 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4239999999999999e-06 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2775 3.1b e108/107 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:23 v 108'605, mlcod=108'601)", 2024-09-16T13:19:04.500 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.916295771000001, 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069960880000000001, 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:04.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562174+0000", 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7000000000000004e-07 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562181+0000", 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2459999999999997e-06 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562194+0000", 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2945000000000001e-05 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562200+0000", 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3679999999999998e-06 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562237+0000", 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7039999999999998e-05 2024-09-16T13:19:04.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562262+0000", 2024-09-16T13:19:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4443999999999998e-05 2024-09-16T13:19:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569096+0000", 2024-09-16T13:19:04.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068345039999999999 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569169+0000", 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2770999999999998e-05 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(3.12 109/108 [PushOp(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22, version: 83'227, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22@83'227, size: 1048576, copy_subset: [0~1048576], clone_subset: {}, snapset: 22=[]:{22=[22]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:19:04.505 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.818160107000001, 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056147790000000003, 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660310+0000", 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-06 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662684+0000", 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023738169999999999 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662716+0000", 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2271999999999999e-05 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662753+0000", 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6851999999999999e-05 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662882+0000", 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012939700000000001 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662892+0000", 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0315e-05 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.665923+0000", 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030311140000000001 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.509 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90825 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch reconnect cookie 140083147222288 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.027455056000001, 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0248692960000001, 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90825 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.510 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451018+0000", 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8569999999999997e-06 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451032+0000", 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4586999999999999e-05 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451078+0000", 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6180000000000002e-05 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451101+0000", 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2773e-05 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451261+0000", 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016003600000000001 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451276+0000", 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4387e-05 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459307+0000", 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080310969999999992 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459318+0000", 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1136e-05 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471312+0000", 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.011993798 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471508+0000", 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019619700000000001 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.475883+0000", 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004375248 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2793 3.1b e110/107 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 v 110'613, mlcod=108'609)", 2024-09-16T13:19:04.515 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.022292234999998, 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070642279999999997, 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:04.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456178+0000", 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.517e-06 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456230+0000", 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2323e-05 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456251+0000", 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0619999999999999e-05 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456269+0000", 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7883000000000001e-05 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456632+0000", 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036355300000000002 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456686+0000", 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3454e-05 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463210+0000", 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065238550000000003 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463241+0000", 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1022999999999998e-05 2024-09-16T13:19:04.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90826 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:19:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:04.520 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.022227978, 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023049584, 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90826 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:04.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456244+0000", 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3000000000000002e-06 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456261+0000", 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7036000000000001e-05 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456294+0000", 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456313+0000", 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8876999999999999e-05 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456444+0000", 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013131300000000001 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456462+0000", 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7920000000000001e-05 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459343+0000", 2024-09-16T13:19:04.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028811539999999999 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459351+0000", 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5870000000000004e-06 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472631+0000", 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0132794970000001 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472737+0000", 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106496 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.473418+0000", 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068136 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.474875+0000", 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001456564 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479175+0000", 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042996450000000004 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479238+0000", 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3610999999999998e-05 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479290+0000", 2024-09-16T13:19:04.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2163999999999997e-05 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1617 3.12 e111/110 3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22 v 111'330, mlcod=0'0)", 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.012498155999999, 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060955660000000002, 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465972+0000", 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2999999999999999e-07 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465984+0000", 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2133999999999999e-05 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465998+0000", 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4184e-05 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466009+0000", 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1371e-05 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467605+0000", 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015953129999999999 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467632+0000", 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7447e-05 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472017+0000", 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043852589999999999 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472066+0000", 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8928000000000003e-05 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.532 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1618 3.12 e111/110 3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22 v 111'332, mlcod=0'0)", 2024-09-16T13:19:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.0119015, 2024-09-16T13:19:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00616418, 2024-09-16T13:19:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466568+0000", 2024-09-16T13:19:04.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6000000000000003e-07 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466575+0000", 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8809999999999998e-06 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466899+0000", 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032458399999999999 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466912+0000", 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2767e-05 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469208+0000", 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022958369999999998 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469231+0000", 2024-09-16T13:19:04.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2578e-05 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472690+0000", 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003459443 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472731+0000", 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1430000000000001e-05 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.537 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1621 3.12 e111/110 3:4a3469d0:::rbd_data.112d12abb20f.000000000000003a:28 v 111'338, mlcod=111'334)", 2024-09-16T13:19:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.989993939000001, 2024-09-16T13:19:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005747411, 2024-09-16T13:19:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.538 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488476+0000", 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488497+0000", 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1542000000000001e-05 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488508+0000", 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0461e-05 2024-09-16T13:19:04.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488514+0000", 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2990000000000002e-06 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488568+0000", 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4559999999999999e-05 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488591+0000", 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2416e-05 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494170+0000", 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055793279999999997 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494222+0000", 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2079000000000001e-05 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.542 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90830 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.989393295999999, 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054188910000000003, 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90830 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489075+0000", 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489077+0000", 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.544e-06 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489084+0000", 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5919999999999997e-06 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489111+0000", 2024-09-16T13:19:04.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7484999999999999e-05 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489140+0000", 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8466000000000002e-05 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489192+0000", 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1946e-05 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489255+0000", 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3751000000000007e-05 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489369+0000", 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011379000000000001 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494232+0000", 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048629679999999996 2024-09-16T13:19:04.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494426+0000", 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000194258 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494467+0000", 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0339999999999997e-05 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494494+0000", 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7750999999999999e-05 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1622 3.12 e111/110 3:498ca772:::rbd_data.112d12abb20f.0000000000000017:28 v 111'340, mlcod=111'334)", 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.988974284000001, 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074768710000000004, 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.549 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489503+0000", 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8720000000000002e-06 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489515+0000", 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2191000000000001e-05 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489524+0000", 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6170000000000007e-06 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489592+0000", 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7670999999999996e-05 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489614+0000", 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2112000000000001e-05 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496931+0000", 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073173739999999998 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496971+0000", 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0247999999999998e-05 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.553 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90832 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch unwatch cookie 140083147222288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.983681540999999, 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055530110000000001, 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.554 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90832 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494788+0000", 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3900000000000004e-07 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494791+0000", 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9050000000000001e-06 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494799+0000", 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3100000000000001e-06 2024-09-16T13:19:04.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494804+0000", 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2170000000000002e-06 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496030+0000", 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012255689999999999 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496148+0000", 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117612 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496408+0000", 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026060999999999999 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:04.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.499840+0000", 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034319670000000002 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500173+0000", 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033327399999999999 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500232+0000", 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8303000000000001e-05 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500340+0000", 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010860499999999999 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:91826 3.11 3:8c261a44:::rbd_data.112d770f4ae2.000000000000000e:head [delete] snapc 1c=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.831897397000001, 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077560520000000003, 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.560 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 91826 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646576+0000", 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2699999999999998e-06 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646592+0000", 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6439999999999998e-05 2024-09-16T13:19:04.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646637+0000", 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5120000000000002e-05 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646657+0000", 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9737000000000001e-05 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646782+0000", 2024-09-16T13:19:04.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012466699999999999 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646790+0000", 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6680000000000007e-06 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651149+0000", 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043590959999999998 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651538+0000", 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000389071 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651714+0000", 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017587200000000001 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654242+0000", 2024-09-16T13:19:04.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025272659999999998 2024-09-16T13:19:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654249+0000", 2024-09-16T13:19:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.61e-06 2024-09-16T13:19:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654294+0000", 2024-09-16T13:19:04.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4644000000000001e-05 2024-09-16T13:19:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654327+0000", 2024-09-16T13:19:04.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3590999999999997e-05 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92599 3.11 3:8b1d31ff:::rbd_data.112d3e228c72.0000000000000004:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.8390969659999996, 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070407359999999997, 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.569 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92599 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639375+0000", 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9399999999999998e-06 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639386+0000", 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1606999999999999e-05 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639426+0000", 2024-09-16T13:19:04.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9329000000000002e-05 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639439+0000", 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.348e-05 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639642+0000", 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000202976 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639865+0000", 2024-09-16T13:19:04.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022277500000000001 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640355+0000", 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049025000000000002 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.644293+0000", 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003937739 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646196+0000", 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019030410000000001 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646332+0000", 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013587899999999999 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646413+0000", 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.072e-05 2024-09-16T13:19:04.574 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92601 3.13 3:cbdfcc30:::rbd_data.112d3e228c72.0000000000000006:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.8389953259999992, 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069350899999999997, 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.575 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92601 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639475+0000", 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.59e-06 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639482+0000", 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.46e-06 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639568+0000", 2024-09-16T13:19:04.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6064999999999997e-05 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639591+0000", 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3564e-05 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639945+0000", 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035337399999999999 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.578 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640456+0000", 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051193699999999996 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640736+0000", 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027973199999999998 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.643326+0000", 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025895129999999999 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646146+0000", 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028207359999999999 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646277+0000", 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013046999999999999 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646409+0000", 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013164900000000001 2024-09-16T13:19:04.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92605 3.11 3:8caec98f:::rbd_data.112d3e228c72.000000000000000a:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.8365605550000002, 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067373449999999996, 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.581 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92605 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641911+0000", 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-06 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641925+0000", 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.403e-05 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641959+0000", 2024-09-16T13:19:04.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3448999999999998e-05 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641976+0000", 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7397e-05 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.642102+0000", 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012618899999999999 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.642511+0000", 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040815799999999999 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.643089+0000", 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057817299999999997 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.644631+0000", 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015421969999999999 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648481+0000", 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038498930000000001 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648543+0000", 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2477000000000005e-05 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648646+0000", 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010228199999999999 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.586 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:93417 3.e 3:74f3d54d:::rbd_data.112d30ff5030.000000000000000b:head [delete] snapc 13=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.3813565140000001, 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077949680000000002, 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 93417 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097116+0000", 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3859999999999999e-06 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097141+0000", 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5732e-05 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097191+0000", 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9854000000000001e-05 2024-09-16T13:19:04.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097211+0000", 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0046000000000001e-05 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097304+0000", 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2318999999999995e-05 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097311+0000", 2024-09-16T13:19:04.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4089999999999997e-06 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.101746+0000", 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044344780000000004 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.102087+0000", 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034153299999999998 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.102361+0000", 2024-09-16T13:19:04.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027436799999999999 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104058+0000", 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001696431 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104805+0000", 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0007473 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104848+0000", 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3112999999999999e-05 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104907+0000", 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8999000000000002e-05 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:93579 3.11 3:8cf36ad7:::rbd_data.112d8b6bb15.0000000000000010:head [delete] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-16T13:19:04.593 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:01.130561+0000", 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.3479076230000002, 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0053757259999999999, 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 93579 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130561+0000", 2024-09-16T13:19:04.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130561+0000", 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130563+0000", 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5e-06 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130570+0000", 2024-09-16T13:19:04.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4800000000000004e-06 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130590+0000", 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0023000000000001e-05 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130603+0000", 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2656e-05 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.130752+0000", 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014950799999999999 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.131228+0000", 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00047526099999999999 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.131487+0000", 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025928099999999999 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.134000+0000", 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00251294 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.135812+0000", 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018124739999999999 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.135888+0000", 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6032000000000007e-05 2024-09-16T13:19:04.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:04.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:04.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:04.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:01.135937+0000", 2024-09-16T13:19:04.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8571e-05 2024-09-16T13:19:04.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.599 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:04.599 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:04.599 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:04.601 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:04.595+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:04.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:19:04.702 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:04.695+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:04.709 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:04.709 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:04.709 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:19:04.709 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:04.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:19:04.804 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:04.799+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:04.813 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:04.813 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:04.813 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:19:04.813 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:19:04.813 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:04.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:19:04.905 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:04.897+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:04.927 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66873 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4034560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:19:04.927 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306607810999999, 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034635435999999999, 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66873 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:19:04.928 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615785+0000", 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.929 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615786+0000", 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615792+0000", 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1900000000000003e-06 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615795+0000", 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9450000000000002e-06 2024-09-16T13:19:04.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650232+0000", 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034437669999999997 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650301+0000", 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8329000000000003e-05 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.931 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650420+0000", 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119132 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66874 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4042752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306574605000002, 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034827871000000003, 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.932 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66874 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.933 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615820+0000", 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.063e-06 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.934 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615826+0000", 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9739999999999999e-06 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615835+0000", 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5920000000000001e-06 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650445+0000", 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034610100999999997 2024-09-16T13:19:04.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650512+0000", 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650645+0000", 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013316399999999999 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66875 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4050944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306553696000002, 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032338000000003, 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66875 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.937 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:19:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:19:04.938 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615839+0000", 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615841+0000", 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9250000000000002e-06 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.939 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615846+0000", 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.711e-06 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615851+0000", 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9579999999999998e-06 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650676+0000", 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034824524000000003 2024-09-16T13:19:04.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650758+0000", 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1547999999999999e-05 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650871+0000", 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113244 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66876 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4059136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306536801, 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035239591000000001, 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66876 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.942 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615856+0000", 2024-09-16T13:19:04.943 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615858+0000", 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.857e-06 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615863+0000", 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5389999999999998e-06 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.944 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615866+0000", 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650899+0000", 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032448000000001 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650973+0000", 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3627000000000005e-05 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.945 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651095+0000", 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012239400000000001 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66877 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4067328~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306522100999999, 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035445691000000001, 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.946 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66877 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615872+0000", 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.703e-06 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615877+0000", 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2650000000000004e-06 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615881+0000", 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.292e-06 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651123+0000", 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035242373 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651198+0000", 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5096000000000001e-05 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651316+0000", 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011757 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66878 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4075520~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:19:04.950 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306508207999997, 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035647261999999999, 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66878 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5400000000000002e-07 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615890+0000", 2024-09-16T13:19:04.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6189999999999999e-06 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615896+0000", 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6529999999999999e-06 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615900+0000", 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0069999999999999e-06 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651337+0000", 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035437877999999999 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651395+0000", 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7219999999999998e-05 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651531+0000", 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013653100000000001 2024-09-16T13:19:04.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66879 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306488866999999, 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035838241999999999, 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.955 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66879 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:19:04.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615904+0000", 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615906+0000", 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615911+0000", 2024-09-16T13:19:04.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6679999999999998e-06 2024-09-16T13:19:04.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615914+0000", 2024-09-16T13:19:04.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.151e-06 2024-09-16T13:19:04.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651558+0000", 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035643672000000001 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651624+0000", 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6070999999999998e-05 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651742+0000", 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011729499999999999 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.959 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66880 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4091904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306474287, 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036042148000000003, 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.960 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66880 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2099999999999997e-07 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615924+0000", 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615930+0000", 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6250000000000004e-06 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615933+0000", 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3950000000000001e-06 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651773+0000", 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035839579000000003 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651846+0000", 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2602999999999996e-05 2024-09-16T13:19:04.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651960+0000", 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114319 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66881 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4100096~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306453269999999, 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036242563999999998, 2024-09-16T13:19:04.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66881 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615941+0000", 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.72e-06 2024-09-16T13:19:04.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615946+0000", 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.259e-06 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615950+0000", 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651985+0000", 2024-09-16T13:19:04.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036035821000000003 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652058+0000", 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2651999999999995e-05 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652181+0000", 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123428 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66882 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4108288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306439240000003, 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036437628999999999, 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66882 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.969 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7300000000000002e-07 2024-09-16T13:19:04.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615955+0000", 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8199999999999999e-06 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615961+0000", 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2349999999999996e-06 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615965+0000", 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4620000000000001e-06 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652207+0000", 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036241873000000001 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652276+0000", 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9105999999999999e-05 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652391+0000", 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011476000000000001 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66883 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4116480~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306424016999998, 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036642620000000001, 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.973 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66883 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:19:04.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:19:04.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.77e-07 2024-09-16T13:19:04.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615970+0000", 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7519999999999999e-06 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615977+0000", 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9349999999999997e-06 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615985+0000", 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1389999999999995e-06 2024-09-16T13:19:04.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652416+0000", 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036430381999999997 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652487+0000", 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1698000000000004e-05 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652611+0000", 2024-09-16T13:19:04.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123337 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66884 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4124672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.30615959, 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036613363000000003, 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.978 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66884 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.16e-07 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616237+0000", 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5719999999999999e-06 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616245+0000", 2024-09-16T13:19:04.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8820000000000001e-06 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616251+0000", 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9630000000000003e-06 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652638+0000", 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036387705999999999 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652719+0000", 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0469999999999994e-05 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652846+0000", 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126954 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66885 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4132864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:19:04.982 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306137038000003, 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036785929000000002, 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66885 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0999999999999999e-07 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616258+0000", 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0269999999999998e-06 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616263+0000", 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8760000000000002e-06 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616267+0000", 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652873+0000", 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036606376000000003 2024-09-16T13:19:04.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652956+0000", 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2824000000000005e-05 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653041+0000", 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4918000000000002e-05 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66886 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4141056~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306120608999997, 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078279999999998, 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66886 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:19:04.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616274+0000", 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.73e-06 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616279+0000", 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616283+0000", 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1420000000000002e-06 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653119+0000", 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036836016999999999 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653204+0000", 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5575999999999995e-05 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653350+0000", 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014569499999999999 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66887 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4149248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306106106000001, 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037308557999999999, 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.991 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66887 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.84e-07 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616288+0000", 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7099999999999999e-06 2024-09-16T13:19:04.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616294+0000", 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4029999999999998e-06 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616297+0000", 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0079999999999998e-06 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653375+0000", 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078883 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653455+0000", 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0034e-05 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653595+0000", 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013913599999999999 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:04.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66888 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4157440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306091735000003, 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537323999999997, 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66888 2024-09-16T13:19:04.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:04.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616301+0000", 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616302+0000", 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616308+0000", 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616317+0000", 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2229999999999996e-06 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653620+0000", 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037302572999999999 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653701+0000", 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1771e-05 2024-09-16T13:19:04.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653838+0000", 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013640599999999999 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66889 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4165632~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:19:05.000 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306071207000002, 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037755019000000001, 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66889 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2300000000000002e-07 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616323+0000", 2024-09-16T13:19:05.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.905e-06 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616329+0000", 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4639999999999997e-06 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616332+0000", 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0180000000000002e-06 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653862+0000", 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037529973000000001 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653937+0000", 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4954000000000001e-05 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654076+0000", 2024-09-16T13:19:05.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013928200000000001 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66890 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4173824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.306055800999999, 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037917073000000003, 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.005 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66890 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616337+0000", 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616338+0000", 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616371+0000", 2024-09-16T13:19:05.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2806999999999999e-05 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616375+0000", 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.827e-06 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654115+0000", 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.03773986 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654181+0000", 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5652000000000001e-05 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654253+0000", 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2789999999999999e-05 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.009 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66891 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4182016~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.30600793, 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038160044999999997, 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66891 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:19:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:19:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616385+0000", 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616387+0000", 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.356e-06 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616393+0000", 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9519999999999999e-06 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616396+0000", 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2959999999999999e-06 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654302+0000", 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037905031999999998 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654395+0000", 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3560000000000005e-05 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654544+0000", 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014916500000000001 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66892 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4190208~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.305992119000003, 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038342319999999999, 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.014 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66892 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616407+0000", 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4060000000000002e-06 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616412+0000", 2024-09-16T13:19:05.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-06 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616419+0000", 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.037e-06 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654571+0000", 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038152835000000003 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654625+0000", 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3643000000000001e-05 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654742+0000", 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117348 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.018 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:05.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:19:05.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:19:05.021 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:05.015+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:05.021 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T13:19:05.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T13:19:05.116 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:05.116 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:05.116 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:19:05.117 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:05.121 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:05.115+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:05.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:19:05.222 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:05.215+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:05.235 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:05.235 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:05.236 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:19:05.236 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:19:05.236 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:05.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:19:05.323 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:05.317+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:05.342 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64860 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3899392~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.775157354999997, 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035719773000000003, 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.343 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64860 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.344 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563336+0000", 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563342+0000", 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3240000000000002e-06 2024-09-16T13:19:05.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563345+0000", 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.598973+0000", 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035627631999999999 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599006+0000", 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3495999999999999e-05 2024-09-16T13:19:05.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599054+0000", 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7460999999999998e-05 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64861 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3907584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.775142473999999, 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035815461, 2024-09-16T13:19:05.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64861 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563351+0000", 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.854e-06 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563356+0000", 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5940000000000001e-06 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563360+0000", 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4060000000000001e-06 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599076+0000", 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035716171999999997 2024-09-16T13:19:05.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599116+0000", 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9777999999999998e-05 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599164+0000", 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8307999999999999e-05 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64862 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.775127881000003, 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035954097999999997, 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64862 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563364+0000", 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:19:05.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563365+0000", 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.821e-06 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563371+0000", 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0769999999999998e-06 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563374+0000", 2024-09-16T13:19:05.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3560000000000001e-06 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599187+0000", 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035813153 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599252+0000", 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4873e-05 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599317+0000", 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5438000000000006e-05 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64863 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3923968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.775102326999999, 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036051842000000001, 2024-09-16T13:19:05.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64863 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563392+0000", 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3140000000000002e-06 2024-09-16T13:19:05.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563398+0000", 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.066e-06 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563402+0000", 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599339+0000", 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035937678000000001 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599377+0000", 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599441+0000", 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3380999999999995e-05 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.360 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64864 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3932160~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.775085922999999, 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036187015000000003, 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64864 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563406+0000", 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563411+0000", 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563417+0000", 2024-09-16T13:19:05.363 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563421+0000", 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599458+0000", 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036036870999999998 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.364 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599515+0000", 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7067000000000001e-05 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599592+0000", 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7457000000000001e-05 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64865 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3940352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.365 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.775066533, 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036265720000000001, 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64865 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563427+0000", 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.001e-06 2024-09-16T13:19:05.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563433+0000", 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5489999999999997e-06 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563436+0000", 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.484e-06 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599613+0000", 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036176815000000001 2024-09-16T13:19:05.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599646+0000", 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3192999999999999e-05 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599690+0000", 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4240000000000003e-05 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.369 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64866 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3948544~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.775050755000002, 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036355749999999999, 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64866 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:19:05.370 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563441+0000", 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.371 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563443+0000", 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0870000000000001e-06 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563448+0000", 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.062e-06 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563451+0000", 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.372 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599712+0000", 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036261007999999997 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599753+0000", 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0476999999999999e-05 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599796+0000", 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3226000000000003e-05 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64867 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3956736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.775036014999998, 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036484356000000003, 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64867 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.374 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563456+0000", 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.375 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563458+0000", 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9980000000000002e-06 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563463+0000", 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563466+0000", 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.376 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599810+0000", 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036343683000000002 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599864+0000", 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3565e-05 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599939+0000", 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5889999999999993e-05 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.377 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64868 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3964928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.775020857000001, 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036573319999999999, 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64868 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.378 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563471+0000", 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563473+0000", 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563479+0000", 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.178e-06 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563483+0000", 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6339999999999999e-06 2024-09-16T13:19:05.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599963+0000", 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036480090999999999 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600001+0000", 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8340999999999998e-05 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600044+0000", 2024-09-16T13:19:05.381 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64869 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3973120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.775004883000001, 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036707818000000003, 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.382 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64869 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.383 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563487+0000", 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9100000000000003e-07 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563489+0000", 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563494+0000", 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.384 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563498+0000", 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.174e-06 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600061+0000", 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036563127000000001 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600119+0000", 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8476000000000001e-05 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600194+0000", 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4869000000000004e-05 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64870 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3981312~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.774989841, 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036852528000000002, 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.386 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64870 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:19:05.387 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563502+0000", 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563504+0000", 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9479999999999998e-06 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.388 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563509+0000", 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563512+0000", 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3239999999999999e-06 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600218+0000", 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036705812999999997 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600280+0000", 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2298000000000006e-05 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600354+0000", 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3343000000000006e-05 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64871 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3989504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.390 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.774975134999998, 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036938311000000001, 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64871 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.392 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563518+0000", 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9130000000000001e-06 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563523+0000", 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563529+0000", 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0710000000000002e-06 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.393 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036847958 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600411+0000", 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4388999999999999e-05 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600454+0000", 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3411000000000002e-05 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64872 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3997696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.774958628, 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037077038999999999, 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64872 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.395 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563533+0000", 2024-09-16T13:19:05.396 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563535+0000", 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1270000000000001e-06 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563540+0000", 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3120000000000001e-06 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.397 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563544+0000", 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4699999999999998e-06 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600476+0000", 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036931691000000003 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600543+0000", 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7475e-05 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.398 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600610+0000", 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6397999999999997e-05 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64873 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4005888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.774943602999997, 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037239282999999998, 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.399 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64873 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563550+0000", 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563555+0000", 2024-09-16T13:19:05.401 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563558+0000", 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2009999999999999e-06 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600630+0000", 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037071933000000001 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600702+0000", 2024-09-16T13:19:05.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1245e-05 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600787+0000", 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5290000000000005e-05 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64874 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4014080~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.774929157000003, 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037394846000000002, 2024-09-16T13:19:05.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64874 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0400000000000002e-07 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563565+0000", 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1900000000000002e-06 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563570+0000", 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4199999999999998e-06 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563574+0000", 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600812+0000", 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037237600000000003 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4690000000000006e-05 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600957+0000", 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0463999999999994e-05 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64875 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4022272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:19:05.407 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.774913361999999, 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537883000000001, 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64875 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.65e-07 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563580+0000", 2024-09-16T13:19:05.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563587+0000", 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2589999999999997e-06 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563591+0000", 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600982+0000", 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037390117 2024-09-16T13:19:05.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601044+0000", 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2788999999999998e-05 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601116+0000", 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1282e-05 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64876 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4030464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.774896036999998, 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037682571999999998, 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64876 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563596+0000", 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563613+0000", 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7935999999999999e-05 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563620+0000", 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4740000000000002e-06 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563624+0000", 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601141+0000", 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037517170000000002 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601203+0000", 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2187999999999995e-05 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601278+0000", 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64877 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4038656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.774862427999999, 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037813404000000002, 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64877 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:19:05.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563631+0000", 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1330000000000002e-06 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563637+0000", 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563640+0000", 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601304+0000", 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037664438000000001 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601372+0000", 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7618000000000002e-05 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601442+0000", 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-05 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.419 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64878 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4046848~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.774833254999997, 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037956456, 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64878 2024-09-16T13:19:05.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563659+0000", 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:19:05.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563661+0000", 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-06 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563667+0000", 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3470000000000004e-06 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563671+0000", 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7170000000000002e-06 2024-09-16T13:19:05.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601473+0000", 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037801955999999998 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601537+0000", 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3960000000000004e-05 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601614+0000", 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64879 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4055040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 51.774816616000003, 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038485017000000003, 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64879 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.424 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4099999999999998e-07 2024-09-16T13:19:05.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563677+0000", 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1299999999999999e-06 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563682+0000", 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563688+0000", 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3329999999999996e-06 2024-09-16T13:19:05.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601640+0000", 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037951864000000002 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601705+0000", 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5573000000000004e-05 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.602160+0000", 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000454411 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.427 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.428 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:05.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:19:05.429 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:05.421+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:05.530 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:05.527+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:05.570 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:05.570 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:05.570 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:19:05.570 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:05.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:19:05.631 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:05.625+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:05.681 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:05.682 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:05.682 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:19:05.682 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:19:05.682 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:05.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:19:05.731 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:05.725+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:05.795 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.795 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 121.782318066, 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:19:05.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:05.800 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 121.78115114400001, 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:05.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:19:05.804 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 113.731807378, 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:19:05.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.809 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 113.73119793799999, 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:19:05.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:19:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:19:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:19:05.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.814 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 113.23391412399999, 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:19:05.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:19:05.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 113.232813308, 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:19:05.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.821 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 113.232673668, 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:19:05.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.826 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 113.23217313799999, 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:19:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:19:05.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 108.23700375200001, 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:19:05.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:19:05.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.836 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 97.980589910999996, 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.837 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:19:05.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:19:05.841 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 97.368026803999996, 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:05.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:19:05.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:19:05.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 70.529920634000007, 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:19:05.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 70.165793300999994, 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.850 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:19:05.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.852 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.854 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:75420 3.4 e88/23 3:2605e3b8:::rbd_object_map.112d9326a5dd.0000000000000026:head v 88'165, mlcod=88'163)", 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 46.287064285, 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055875079999999997, 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503433+0000", 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503451+0000", 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7640999999999999e-05 2024-09-16T13:19:05.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503476+0000", 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5599999999999999e-05 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503495+0000", 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.825e-05 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.857 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503611+0000", 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011631799999999999 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503644+0000", 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3476000000000003e-05 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.508954+0000", 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053093799999999998 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.509019+0000", 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5425999999999993e-05 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1195 3.4 e101/23 3:24bef2c8:::rbd_data.112da28a57db.0000000000000020:27 v 101'326, mlcod=99'322)", 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.416850659000001, 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005296243, 2024-09-16T13:19:05.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373646+0000", 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7e-07 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373653+0000", 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4000000000000003e-06 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374730+0000", 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00107654 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374741+0000", 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1474999999999999e-05 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374829+0000", 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7584999999999996e-05 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374854+0000", 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5269000000000002e-05 2024-09-16T13:19:05.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378893+0000", 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040386770000000001 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378942+0000", 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1197 3.4 e101/23 3:26d31b55:::rbd_data.112da28a57db.000000000000000c:27 v 101'328, mlcod=99'322)", 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.415733586000002, 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046376150000000003, 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374763+0000", 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5600000000000005e-07 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374784+0000", 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0781999999999999e-05 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374794+0000", 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0558e-05 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374800+0000", 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375389+0000", 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058970500000000005 2024-09-16T13:19:05.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375407+0000", 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7801000000000001e-05 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379351+0000", 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039435030000000001 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379400+0000", 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9332000000000001e-05 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:89341 3.4 e101/23 3:21ab973d:::rbd_data.112da28a57db.0000000000000002:head v 101'329, mlcod=101'326)", 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 28.293273793000001, 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005014859, 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497223+0000", 2024-09-16T13:19:05.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8599999999999998e-07 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497231+0000", 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6709999999999999e-06 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497245+0000", 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.489e-05 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497253+0000", 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6199999999999999e-06 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498106+0000", 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00085257599999999996 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498130+0000", 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4776000000000001e-05 2024-09-16T13:19:05.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502154+0000", 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004023703 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502237+0000", 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2936999999999996e-05 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.4.0:2718 3.9 e106/23 3:9702159d:::rbd_data.112d404789fd.000000000000001c:22 v 106'440, mlcod=105'437)", 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 23.369455648999999, 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063154409999999998, 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:05.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421041+0000", 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421055+0000", 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4579999999999999e-05 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421080+0000", 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4859e-05 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421089+0000", 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8950000000000004e-06 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421166+0000", 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6892999999999997e-05 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421197+0000", 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1325000000000002e-05 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427304+0000", 2024-09-16T13:19:05.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061062449999999997 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427356+0000", 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2197000000000002e-05 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.4.0:2792 3.9 e110/23 3:938f1173:::rbd_data.112d94eb720f.0000000000000006:20 v 110'460, mlcod=106'457)", 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:19:05.877 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 19.334885481000001, 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041377590000000004, 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2300000000000001e-07 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455621+0000", 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0064000000000001e-05 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455658+0000", 2024-09-16T13:19:05.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6288000000000002e-05 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455678+0000", 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0812999999999999e-05 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455817+0000", 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013879800000000001 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455886+0000", 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8842000000000003e-05 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.459651+0000", 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037649519999999998 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.459748+0000", 2024-09-16T13:19:05.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7379000000000004e-05 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1263 3.4 e119/23 3:2111795e:::rbd_data.112d3e228c72.0000000000000003:18 v 119'349, mlcod=118'345)", 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:55.540473+0000", 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 10.250022655, 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039574179999999999, 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:05.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540473+0000", 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540473+0000", 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540474+0000", 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:05.883 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540484+0000", 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0479000000000001e-05 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540494+0000", 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3910000000000004e-06 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540501+0000", 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7950000000000008e-06 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540593+0000", 2024-09-16T13:19:05.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1841000000000004e-05 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540622+0000", 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8597000000000001e-05 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.544386+0000", 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003763944 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.544431+0000", 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5039e-05 2024-09-16T13:19:05.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:05.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:05.886 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:05.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:19:05.888 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:05.885+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:05.988 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:05.979+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:06.029 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:19:06.032 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T13:19:06.045 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T13:19:06.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:19:06.051 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:06.052 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:06.052 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:19:06.052 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:06.062 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:19:06.106 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:06.101+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:06.159 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:06.159 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:06.159 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:19:06.159 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:19:06.159 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:06.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:19:06.189 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:06.183+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:06.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62278 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947720009000001, 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027907498999999999, 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62278 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:19:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:19:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:19:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.270 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315829+0000", 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.79e-06 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315835+0000", 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.682e-06 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315838+0000", 2024-09-16T13:19:06.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0730000000000001e-06 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343640+0000", 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027802384999999999 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343683+0000", 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3058e-05 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343734+0000", 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1104999999999999e-05 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947705448000001, 2024-09-16T13:19:06.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:19:06.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:19:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:19:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:19:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947681482, 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.278 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:19:06.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947666312000003, 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.283 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:19:06.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:06.287 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947651888999999, 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:19:06.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.291 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947630644, 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:19:06.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947616074999999, 2024-09-16T13:19:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:19:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:19:06.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947597305999999, 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:19:06.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947582410999999, 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:19:06.309 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.310 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:19:06.311 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947568199999999, 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.312 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:06.313 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947554830999998, 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.317 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.319 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:19:06.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947540148999998, 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.323 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:19:06.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947524979999997, 2024-09-16T13:19:06.326 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:06.327 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:19:06.329 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.330 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947510340000001, 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.331 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:19:06.333 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:19:06.334 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947479612000002, 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:19:06.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:19:06.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:19:06.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.339 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947458595000001, 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.340 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.343 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947438486999999, 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.344 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.345 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:19:06.347 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947418583999998, 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.349 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:19:06.350 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.351 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947403371, 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.354 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.355 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:19:06.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 52.947387837000001, 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.358 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:19:06.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:06.360 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:06.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:06.362 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:06.362 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:19:06.362 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:19:06.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:06.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:06.362 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:06.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:19:06.366 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:06.368 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:19:06.368 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-16T13:19:06.015156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":244198106,"num_objects":690,"num_object_clones":146,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":331,"num_read":140805,"num_read_kb":678295,"num_write":18671,"num_write_kb":297182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":865,"num_bytes_recovered":44298100,"num_keys_recovered":148,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15608,"ondisk_log_size":15608,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2454884,"kb_used_data":28080,"kb_used_omap":62,"kb_used_meta":2426497,"kb_avail":752519836,"statfs":{"total":773094113280,"available":770580312064,"internally_reserved":0,"allocated":28753920,"data_stored":119406167,"data_compressed":7858208,"data_compressed_allocated":9228288,"data_compressed_original":107239862,"omap_allocated":63650,"internal_metadata":2484733790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-174328235,"num_objects":-142,"num_object_clones":-22,"num_object_copies":-284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-142,"num_whiteouts":-19,"num_read":1391,"num_read_kb":1191,"num_write":270,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030411"},"pg_stats":[{"pgid":"3.1f","version":"127'338","reported_seq":1780,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.984517+0000","last_change":"2024-09-16T13:19:02.984823+0000","last_active":"2024-09-16T13:19:02.984517+0000","last_peered":"2024-09-16T13:19:02.984517+0000","last_clean":"2024-09-16T13:19:02.984517+0000","last_became_active":"2024-09-16T13:18:03.119735+0000","last_became_peered":"2024-09-16T13:18:03.119735+0000","last_unstale":"2024-09-16T13:19:02.984517+0000","last_undegraded":"2024-09-16T13:19:02.984517+0000","last_fullsized":"2024-09-16T13:19:02.984517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:02.061478+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0054926480000000001,"stat_sum":{"num_bytes":11337927,"num_objects":16,"num_object_clones":3,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":8,"num_read":2043,"num_read_kb":15887,"num_write":340,"num_write_kb":5580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":922127,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.1e","version":"126'768","reported_seq":2408,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.001925+0000","last_change":"2024-09-16T13:19:04.002399+0000","last_active":"2024-09-16T13:19:04.001925+0000","last_peered":"2024-09-16T13:19:04.001925+0000","last_clean":"2024-09-16T13:19:04.001925+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:19:04.001925+0000","last_undegraded":"2024-09-16T13:19:04.001925+0000","last_fullsized":"2024-09-16T13:19:04.001925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:50.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001766447,"stat_sum":{"num_bytes":4454089,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":9,"num_read":5463,"num_read_kb":7000,"num_write":964,"num_write_kb":6235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.1d","version":"127'424","reported_seq":2559,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.178506+0000","last_change":"2024-09-16T13:19:02.979901+0000","last_active":"2024-09-16T13:19:03.178506+0000","last_peered":"2024-09-16T13:19:03.178506+0000","last_clean":"2024-09-16T13:19:03.178506+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:19:03.178506+0000","last_undegraded":"2024-09-16T13:19:03.178506+0000","last_fullsized":"2024-09-16T13:19:03.178506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:14.387827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000638585,"stat_sum":{"num_bytes":2584403,"num_objects":22,"num_object_clones":4,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":11,"num_read":4763,"num_read_kb":13863,"num_write":438,"num_write_kb":6042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.1c","version":"128'421","reported_seq":2583,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.158121+0000","last_change":"2024-09-16T13:19:04.021435+0000","last_active":"2024-09-16T13:19:04.158121+0000","last_peered":"2024-09-16T13:19:04.158121+0000","last_clean":"2024-09-16T13:19:04.158121+0000","last_became_active":"2024-09-16T13:17:10.061217+0000","last_became_peered":"2024-09-16T13:17:10.061217+0000","last_unstale":"2024-09-16T13:19:04.158121+0000","last_undegraded":"2024-09-16T13:19:04.158121+0000","last_fullsized":"2024-09-16T13:19:04.158121+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:10.571278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072099600000000001,"stat_sum":{"num_bytes":3608797,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":9,"num_read":2940,"num_read_kb":25156,"num_write":527,"num_write_kb":12813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.1b","version":"127'628","reported_seq":5473,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.000656+0000","last_change":"2024-09-16T13:19:04.001325+0000","last_active":"2024-09-16T13:19:04.000656+0000","last_peered":"2024-09-16T13:19:04.000656+0000","last_clean":"2024-09-16T13:19:04.000656+0000","last_became_active":"2024-09-16T13:18:44.442336+0000","last_became_peered":"2024-09-16T13:18:44.442336+0000","last_unstale":"2024-09-16T13:19:04.000656+0000","last_undegraded":"2024-09-16T13:19:04.000656+0000","last_fullsized":"2024-09-16T13:19:04.000656+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:34.313429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001845232,"stat_sum":{"num_bytes":3148182,"num_objects":31,"num_object_clones":7,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":15,"num_read":6522,"num_read_kb":28075,"num_write":890,"num_write_kb":17338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2924073,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.1a","version":"128'636","reported_seq":7784,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.180571+0000","last_change":"2024-09-16T13:19:04.000062+0000","last_active":"2024-09-16T13:19:04.180571+0000","last_peered":"2024-09-16T13:19:04.180571+0000","last_clean":"2024-09-16T13:19:04.180571+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:19:04.180571+0000","last_undegraded":"2024-09-16T13:19:04.180571+0000","last_fullsized":"2024-09-16T13:19:04.180571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:31.071219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012938120000000001,"stat_sum":{"num_bytes":14964474,"num_objects":20,"num_object_clones":5,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":10,"num_read":9157,"num_read_kb":54961,"num_write":840,"num_write_kb":18791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.19","version":"128'589","reported_seq":5404,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.009499+0000","last_change":"2024-09-16T13:19:04.009747+0000","last_active":"2024-09-16T13:19:04.009499+0000","last_peered":"2024-09-16T13:19:04.009499+0000","last_clean":"2024-09-16T13:19:04.009499+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:19:04.009499+0000","last_undegraded":"2024-09-16T13:19:04.009499+0000","last_fullsized":"2024-09-16T13:19:04.009499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:42:30.327300+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0072239230000000001,"stat_sum":{"num_bytes":5689078,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":12,"num_read":7637,"num_read_kb":32040,"num_write":805,"num_write_kb":13589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.18","version":"128'414","reported_seq":2317,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.006692+0000","last_change":"2024-09-16T13:19:04.007967+0000","last_active":"2024-09-16T13:19:04.006692+0000","last_peered":"2024-09-16T13:19:04.006692+0000","last_clean":"2024-09-16T13:19:04.006692+0000","last_became_active":"2024-09-16T13:17:02.962736+0000","last_became_peered":"2024-09-16T13:17:02.962736+0000","last_unstale":"2024-09-16T13:19:04.006692+0000","last_undegraded":"2024-09-16T13:19:04.006692+0000","last_fullsized":"2024-09-16T13:19:04.006692+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:27:05.338093+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0089036640000000004,"stat_sum":{"num_bytes":2699245,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":6,"num_read":2801,"num_read_kb":8702,"num_write":535,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.17","version":"126'523","reported_seq":2175,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.965129+0000","last_change":"2024-09-16T13:19:02.965403+0000","last_active":"2024-09-16T13:19:02.965129+0000","last_peered":"2024-09-16T13:19:02.965129+0000","last_clean":"2024-09-16T13:19:02.965129+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:19:02.965129+0000","last_undegraded":"2024-09-16T13:19:02.965129+0000","last_fullsized":"2024-09-16T13:19:02.965129+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:08:57.225261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473364,"stat_sum":{"num_bytes":4912227,"num_objects":24,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":11,"num_read":3492,"num_read_kb":13647,"num_write":563,"num_write_kb":7797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.16","version":"126'398","reported_seq":3007,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.980531+0000","last_change":"2024-09-16T13:19:02.981018+0000","last_active":"2024-09-16T13:19:02.980531+0000","last_peered":"2024-09-16T13:19:02.980531+0000","last_clean":"2024-09-16T13:19:02.980531+0000","last_became_active":"2024-09-16T13:18:44.441740+0000","last_became_peered":"2024-09-16T13:18:44.441740+0000","last_unstale":"2024-09-16T13:19:02.980531+0000","last_undegraded":"2024-09-16T13:19:02.980531+0000","last_fullsized":"2024-09-16T13:19:02.980531+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:20.994894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076613699999999996,"stat_sum":{"num_bytes":5597317,"num_objects":15,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":7,"num_read":5473,"num_read_kb":21309,"num_write":353,"num_write_kb":5655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1185011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.15","version":"126'212","reported_seq":724,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.124662+0000","last_change":"2024-09-16T13:19:02.119706+0000","last_active":"2024-09-16T13:19:02.124662+0000","last_peered":"2024-09-16T13:19:02.124662+0000","last_clean":"2024-09-16T13:19:02.124662+0000","last_became_active":"2024-09-16T13:18:44.435407+0000","last_became_peered":"2024-09-16T13:18:44.435407+0000","last_unstale":"2024-09-16T13:19:02.124662+0000","last_undegraded":"2024-09-16T13:19:02.124662+0000","last_fullsized":"2024-09-16T13:19:02.124662+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:11.546854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031633669999999998,"stat_sum":{"num_bytes":8133452,"num_objects":23,"num_object_clones":7,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":13,"num_read":3249,"num_read_kb":15295,"num_write":187,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1086312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]},{"pgid":"3.14","version":"126'296","reported_seq":4974,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.999237+0000","last_change":"2024-09-16T13:19:04.000058+0000","last_active":"2024-09-16T13:19:03.999237+0000","last_peered":"2024-09-16T13:19:03.999237+0000","last_clean":"2024-09-16T13:19:03.999237+0000","last_became_active":"2024-09-16T13:18:47.462532+0000","last_became_peered":"2024-09-16T13:18:47.462532+0000","last_unstale":"2024-09-16T13:19:03.999237+0000","last_undegraded":"2024-09-16T13:19:03.999237+0000","last_fullsized":"2024-09-16T13:19:03.999237+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:16.599784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288242,"stat_sum":{"num_bytes":2990273,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":5,"num_read":4733,"num_read_kb":25384,"num_write":343,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1053570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.7","version":"0'0","reported_seq":137,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.997965+0000","last_change":"2024-09-16T13:18:44.438193+0000","last_active":"2024-09-16T13:19:02.997965+0000","last_peered":"2024-09-16T13:19:02.997965+0000","last_clean":"2024-09-16T13:19:02.997965+0000","last_became_active":"2024-09-16T13:18:44.437778+0000","last_became_peered":"2024-09-16T13:18:44.437778+0000","last_unstale":"2024-09-16T13:19:02.997965+0000","last_undegraded":"2024-09-16T13:19:02.997965+0000","last_fullsized":"2024-09-16T13:19:02.997965+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:34.121552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"127'665","reported_seq":5541,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.965762+0000","last_change":"2024-09-16T13:19:02.965762+0000","last_active":"2024-09-16T13:19:02.965762+0000","last_peered":"2024-09-16T13:19:02.965762+0000","last_clean":"2024-09-16T13:19:02.965762+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:19:02.965762+0000","last_undegraded":"2024-09-16T13:19:02.965762+0000","last_fullsized":"2024-09-16T13:19:02.965762+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:52:05.144800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001692762,"stat_sum":{"num_bytes":5398686,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":11,"num_read":6650,"num_read_kb":24623,"num_write":934,"num_write_kb":11031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"2.6","version":"0'0","reported_seq":122,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.984554+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:19:02.984554+0000","last_peered":"2024-09-16T13:19:02.984554+0000","last_clean":"2024-09-16T13:19:02.984554+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:19:02.984554+0000","last_undegraded":"2024-09-16T13:19:02.984554+0000","last_fullsized":"2024-09-16T13:19:02.984554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"128'290","reported_seq":1378,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.007430+0000","last_change":"2024-09-16T13:19:04.008193+0000","last_active":"2024-09-16T13:19:04.007430+0000","last_peered":"2024-09-16T13:19:04.007430+0000","last_clean":"2024-09-16T13:19:04.007430+0000","last_became_active":"2024-09-16T13:18:47.463039+0000","last_became_peered":"2024-09-16T13:18:47.463039+0000","last_unstale":"2024-09-16T13:19:04.007430+0000","last_undegraded":"2024-09-16T13:19:04.007430+0000","last_fullsized":"2024-09-16T13:19:04.007430+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:05:53.068034+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0073940910000000002,"stat_sum":{"num_bytes":5070848,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":13,"num_read":1655,"num_read_kb":8730,"num_write":267,"num_write_kb":5747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1061466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.5","version":"0'0","reported_seq":102,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T13:19:00.994174+0000","last_change":"2024-09-16T13:18:44.435471+0000","last_active":"2024-09-16T13:19:00.994174+0000","last_peered":"2024-09-16T13:19:00.994174+0000","last_clean":"2024-09-16T13:19:00.994174+0000","last_became_active":"2024-09-16T13:18:44.434851+0000","last_became_peered":"2024-09-16T13:18:44.434851+0000","last_unstale":"2024-09-16T13:19:00.994174+0000","last_undegraded":"2024-09-16T13:19:00.994174+0000","last_fullsized":"2024-09-16T13:19:00.994174+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:12.980758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"127'360","reported_seq":4594,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.002823+0000","last_change":"2024-09-16T13:19:04.003241+0000","last_active":"2024-09-16T13:19:04.002823+0000","last_peered":"2024-09-16T13:19:04.002823+0000","last_clean":"2024-09-16T13:19:04.002823+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:19:04.002823+0000","last_undegraded":"2024-09-16T13:19:04.002823+0000","last_fullsized":"2024-09-16T13:19:04.002823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:30.996264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059763199999999996,"stat_sum":{"num_bytes":1355248,"num_objects":16,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":6,"num_read":4693,"num_read_kb":26876,"num_write":414,"num_write_kb":8111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.4","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.984788+0000","last_change":"2024-09-16T13:17:42.960492+0000","last_active":"2024-09-16T13:19:02.984788+0000","last_peered":"2024-09-16T13:19:02.984788+0000","last_clean":"2024-09-16T13:19:02.984788+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:19:02.984788+0000","last_undegraded":"2024-09-16T13:19:02.984788+0000","last_fullsized":"2024-09-16T13:19:02.984788+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:35:45.152557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"126'511","reported_seq":3717,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.979982+0000","last_change":"2024-09-16T13:19:02.980468+0000","last_active":"2024-09-16T13:19:02.979982+0000","last_peered":"2024-09-16T13:19:02.979982+0000","last_clean":"2024-09-16T13:19:02.979982+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:19:02.979982+0000","last_undegraded":"2024-09-16T13:19:02.979982+0000","last_fullsized":"2024-09-16T13:19:02.979982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:34.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00110391,"stat_sum":{"num_bytes":12552397,"num_objects":33,"num_object_clones":6,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":17,"num_read":5910,"num_read_kb":14107,"num_write":521,"num_write_kb":4259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.3","version":"128'435","reported_seq":2249,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.207566+0000","last_change":"2024-09-16T13:19:04.000634+0000","last_active":"2024-09-16T13:19:04.207566+0000","last_peered":"2024-09-16T13:19:04.207566+0000","last_clean":"2024-09-16T13:19:04.207566+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:19:04.207566+0000","last_undegraded":"2024-09-16T13:19:04.207566+0000","last_fullsized":"2024-09-16T13:19:04.207566+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:44:16.402879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001574317,"stat_sum":{"num_bytes":2465952,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":11,"num_read":2497,"num_read_kb":21118,"num_write":430,"num_write_kb":8994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.2","version":"21'2","reported_seq":123,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.115292+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:19:02.115292+0000","last_peered":"2024-09-16T13:19:02.115292+0000","last_clean":"2024-09-16T13:19:02.115292+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:19:02.115292+0000","last_undegraded":"2024-09-16T13:19:02.115292+0000","last_fullsized":"2024-09-16T13:19:02.115292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"127'408","reported_seq":1687,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.003324+0000","last_change":"2024-09-16T13:19:04.004813+0000","last_active":"2024-09-16T13:19:04.003324+0000","last_peered":"2024-09-16T13:19:04.003324+0000","last_clean":"2024-09-16T13:19:04.003324+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:19:04.003324+0000","last_undegraded":"2024-09-16T13:19:04.003324+0000","last_fullsized":"2024-09-16T13:19:04.003324+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:29.090161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001088489,"stat_sum":{"num_bytes":6388051,"num_objects":24,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":13,"num_read":3120,"num_read_kb":11028,"num_write":469,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.1","version":"0'0","reported_seq":121,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.133127+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:19:02.133127+0000","last_peered":"2024-09-16T13:19:02.133127+0000","last_clean":"2024-09-16T13:19:02.133127+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:19:02.133127+0000","last_undegraded":"2024-09-16T13:19:02.133127+0000","last_fullsized":"2024-09-16T13:19:02.133127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"126'678","reported_seq":2095,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.965061+0000","last_change":"2024-09-16T13:19:02.965479+0000","last_active":"2024-09-16T13:19:02.965061+0000","last_peered":"2024-09-16T13:19:02.965061+0000","last_clean":"2024-09-16T13:19:02.965061+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:19:02.965061+0000","last_undegraded":"2024-09-16T13:19:02.965061+0000","last_fullsized":"2024-09-16T13:19:02.965061+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:03.357310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015907569999999999,"stat_sum":{"num_bytes":13625373,"num_objects":28,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":16,"num_read":3850,"num_read_kb":31161,"num_write":947,"num_write_kb":16370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"2.0","version":"0'0","reported_seq":89,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T13:19:00.994124+0000","last_change":"2024-09-16T13:18:44.435672+0000","last_active":"2024-09-16T13:19:00.994124+0000","last_peered":"2024-09-16T13:19:00.994124+0000","last_clean":"2024-09-16T13:19:00.994124+0000","last_became_active":"2024-09-16T13:18:44.435188+0000","last_became_peered":"2024-09-16T13:18:44.435188+0000","last_unstale":"2024-09-16T13:19:00.994124+0000","last_undegraded":"2024-09-16T13:19:00.994124+0000","last_fullsized":"2024-09-16T13:19:00.994124+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:35.400164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"126'453","reported_seq":2034,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.999594+0000","last_change":"2024-09-16T13:19:04.000215+0000","last_active":"2024-09-16T13:19:03.999594+0000","last_peered":"2024-09-16T13:19:03.999594+0000","last_clean":"2024-09-16T13:19:03.999594+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:19:03.999594+0000","last_undegraded":"2024-09-16T13:19:03.999594+0000","last_fullsized":"2024-09-16T13:19:03.999594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:45.154984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085824899999999995,"stat_sum":{"num_bytes":1800829,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":13,"num_read":3321,"num_read_kb":15348,"num_write":485,"num_write_kb":6255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.3","version":"19'1","reported_seq":122,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.115273+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:19:02.115273+0000","last_peered":"2024-09-16T13:19:02.115273+0000","last_clean":"2024-09-16T13:19:02.115273+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:19:02.115273+0000","last_undegraded":"2024-09-16T13:19:02.115273+0000","last_fullsized":"2024-09-16T13:19:02.115273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":202,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T13:19:00.994294+0000","last_change":"2024-09-16T13:18:44.437802+0000","last_active":"2024-09-16T13:19:00.994294+0000","last_peered":"2024-09-16T13:19:00.994294+0000","last_clean":"2024-09-16T13:19:00.994294+0000","last_became_active":"2024-09-16T13:18:44.437418+0000","last_became_peered":"2024-09-16T13:18:44.437418+0000","last_unstale":"2024-09-16T13:19:00.994294+0000","last_undegraded":"2024-09-16T13:19:00.994294+0000","last_fullsized":"2024-09-16T13:19:00.994294+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:29.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"122'578","reported_seq":1192,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.001395+0000","last_change":"2024-09-16T13:19:04.001798+0000","last_active":"2024-09-16T13:19:04.001395+0000","last_peered":"2024-09-16T13:19:04.001395+0000","last_clean":"2024-09-16T13:19:04.001395+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:19:04.001395+0000","last_undegraded":"2024-09-16T13:19:04.001395+0000","last_fullsized":"2024-09-16T13:19:04.001395+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":578,"log_dups_size":0,"ondisk_log_size":578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:59.515115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022672669999999999,"stat_sum":{"num_bytes":10415520,"num_objects":20,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":6,"num_read":1983,"num_read_kb":19347,"num_write":765,"num_write_kb":12166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.9","version":"126'477","reported_seq":2812,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.166545+0000","last_change":"2024-09-16T13:19:04.000549+0000","last_active":"2024-09-16T13:19:04.166545+0000","last_peered":"2024-09-16T13:19:04.166545+0000","last_clean":"2024-09-16T13:19:04.166545+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:19:04.166545+0000","last_undegraded":"2024-09-16T13:19:04.166545+0000","last_fullsized":"2024-09-16T13:19:04.166545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:49:10.301550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017042649999999999,"stat_sum":{"num_bytes":16858086,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":14,"num_read":4015,"num_read_kb":31205,"num_write":422,"num_write_kb":12345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.a","version":"128'908","reported_seq":2908,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.118957+0000","last_change":"2024-09-16T13:19:04.021081+0000","last_active":"2024-09-16T13:19:04.118957+0000","last_peered":"2024-09-16T13:19:04.118957+0000","last_clean":"2024-09-16T13:19:04.118957+0000","last_became_active":"2024-09-16T13:16:52.299458+0000","last_became_peered":"2024-09-16T13:16:52.299458+0000","last_unstale":"2024-09-16T13:19:04.118957+0000","last_undegraded":"2024-09-16T13:19:04.118957+0000","last_fullsized":"2024-09-16T13:19:04.118957+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":908,"log_dups_size":0,"ondisk_log_size":908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:34.071306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038801599999999998,"stat_sum":{"num_bytes":6322236,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":8,"num_read":4714,"num_read_kb":19938,"num_write":1249,"num_write_kb":12318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.b","version":"128'1072","reported_seq":4160,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.209036+0000","last_change":"2024-09-16T13:19:04.001302+0000","last_active":"2024-09-16T13:19:04.209036+0000","last_peered":"2024-09-16T13:19:04.209036+0000","last_clean":"2024-09-16T13:19:04.209036+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:19:04.209036+0000","last_undegraded":"2024-09-16T13:19:04.209036+0000","last_fullsized":"2024-09-16T13:19:04.209036+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:20.684083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018441429999999999,"stat_sum":{"num_bytes":1850738,"num_objects":19,"num_object_clones":2,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":6,"num_read":9495,"num_read_kb":27050,"num_write":1643,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.c","version":"124'286","reported_seq":1263,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.979265+0000","last_change":"2024-09-16T13:19:02.979820+0000","last_active":"2024-09-16T13:19:02.979265+0000","last_peered":"2024-09-16T13:19:02.979265+0000","last_clean":"2024-09-16T13:19:02.979265+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:19:02.979265+0000","last_undegraded":"2024-09-16T13:19:02.979265+0000","last_fullsized":"2024-09-16T13:19:02.979265+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:39.937083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080311499999999997,"stat_sum":{"num_bytes":9267082,"num_objects":17,"num_object_clones":3,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":1274,"num_read_kb":14888,"num_write":268,"num_write_kb":7028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.d","version":"126'359","reported_seq":2304,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.263694+0000","last_change":"2024-09-16T13:19:02.120551+0000","last_active":"2024-09-16T13:19:02.263694+0000","last_peered":"2024-09-16T13:19:02.263694+0000","last_clean":"2024-09-16T13:19:02.263694+0000","last_became_active":"2024-09-16T13:18:47.468001+0000","last_became_peered":"2024-09-16T13:18:47.468001+0000","last_unstale":"2024-09-16T13:19:02.263694+0000","last_undegraded":"2024-09-16T13:19:02.263694+0000","last_fullsized":"2024-09-16T13:19:02.263694+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:11:32.886561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002450308,"stat_sum":{"num_bytes":20989652,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":14,"num_read":2646,"num_read_kb":18369,"num_write":325,"num_write_kb":14195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]},{"pgid":"3.e","version":"126'502","reported_seq":6995,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.120711+0000","last_change":"2024-09-16T13:19:02.118898+0000","last_active":"2024-09-16T13:19:02.120711+0000","last_peered":"2024-09-16T13:19:02.120711+0000","last_clean":"2024-09-16T13:19:02.120711+0000","last_became_active":"2024-09-16T13:18:47.467028+0000","last_became_peered":"2024-09-16T13:18:47.467028+0000","last_unstale":"2024-09-16T13:19:02.120711+0000","last_undegraded":"2024-09-16T13:19:02.120711+0000","last_fullsized":"2024-09-16T13:19:02.120711+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:35.852378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023549669999999999,"stat_sum":{"num_bytes":2653579,"num_objects":15,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":9,"num_read":8102,"num_read_kb":29094,"num_write":566,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1715710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]},{"pgid":"3.f","version":"125'268","reported_seq":867,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.120097+0000","last_change":"2024-09-16T13:19:02.120441+0000","last_active":"2024-09-16T13:19:02.120097+0000","last_peered":"2024-09-16T13:19:02.120097+0000","last_clean":"2024-09-16T13:19:02.120097+0000","last_became_active":"2024-09-16T13:18:44.444025+0000","last_became_peered":"2024-09-16T13:18:44.444025+0000","last_unstale":"2024-09-16T13:19:02.120097+0000","last_undegraded":"2024-09-16T13:19:02.120097+0000","last_fullsized":"2024-09-16T13:19:02.120097+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:25:03.521529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001780516,"stat_sum":{"num_bytes":39638841,"num_objects":39,"num_object_clones":8,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":16,"num_read":658,"num_read_kb":10363,"num_write":256,"num_write_kb":7710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":63719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]},{"pgid":"3.10","version":"126'245","reported_seq":1304,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.020803+0000","last_change":"2024-09-16T13:19:04.021059+0000","last_active":"2024-09-16T13:19:04.020803+0000","last_peered":"2024-09-16T13:19:04.020803+0000","last_clean":"2024-09-16T13:19:04.020803+0000","last_became_active":"2024-09-16T13:17:42.961695+0000","last_became_peered":"2024-09-16T13:17:42.961695+0000","last_unstale":"2024-09-16T13:19:04.020803+0000","last_undegraded":"2024-09-16T13:19:04.020803+0000","last_fullsized":"2024-09-16T13:19:04.020803+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:35.675244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038742900000000001,"stat_sum":{"num_bytes":9430584,"num_objects":23,"num_object_clones":7,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":10,"num_read":784,"num_read_kb":16305,"num_write":251,"num_write_kb":11082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.11","version":"126'545","reported_seq":2371,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.136740+0000","last_change":"2024-09-16T13:19:02.115710+0000","last_active":"2024-09-16T13:19:02.136740+0000","last_peered":"2024-09-16T13:19:02.136740+0000","last_clean":"2024-09-16T13:19:02.136740+0000","last_became_active":"2024-09-16T13:18:47.462561+0000","last_became_peered":"2024-09-16T13:18:47.462561+0000","last_unstale":"2024-09-16T13:19:02.136740+0000","last_undegraded":"2024-09-16T13:19:02.136740+0000","last_fullsized":"2024-09-16T13:19:02.136740+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":545,"log_dups_size":0,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.079652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080268500000000003,"stat_sum":{"num_bytes":3317604,"num_objects":17,"num_object_clones":3,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":9,"num_read":3854,"num_read_kb":23841,"num_write":727,"num_write_kb":11756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]},{"pgid":"3.12","version":"123'390","reported_seq":7415,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.965549+0000","last_change":"2024-09-16T13:19:02.966110+0000","last_active":"2024-09-16T13:19:02.965549+0000","last_peered":"2024-09-16T13:19:02.965549+0000","last_clean":"2024-09-16T13:19:02.965549+0000","last_became_active":"2024-09-16T13:18:47.462412+0000","last_became_peered":"2024-09-16T13:18:47.462412+0000","last_unstale":"2024-09-16T13:19:02.965549+0000","last_undegraded":"2024-09-16T13:19:02.965549+0000","last_fullsized":"2024-09-16T13:19:02.965549+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:50:25.256657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017562820000000001,"stat_sum":{"num_bytes":1572033,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":5,"num_read":8501,"num_read_kb":35036,"num_write":358,"num_write_kb":9220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2494648,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.13","version":"126'456","reported_seq":1851,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.232677+0000","last_change":"2024-09-16T13:19:02.116504+0000","last_active":"2024-09-16T13:19:02.232677+0000","last_peered":"2024-09-16T13:19:02.232677+0000","last_clean":"2024-09-16T13:19:02.232677+0000","last_became_active":"2024-09-16T13:18:44.443832+0000","last_became_peered":"2024-09-16T13:18:44.443832+0000","last_unstale":"2024-09-16T13:19:02.232677+0000","last_undegraded":"2024-09-16T13:19:02.232677+0000","last_fullsized":"2024-09-16T13:19:02.232677+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:45.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080056599999999995,"stat_sum":{"num_bytes":2514916,"num_objects":26,"num_object_clones":5,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":13,"num_read":4744,"num_read_kb":18494,"num_write":486,"num_write_kb":7694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1653183,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":243607719,"num_objects":686,"num_object_clones":146,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":331,"num_read":140739,"num_read_kb":678240,"num_write":18568,"num_write_kb":295552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":863,"num_bytes_recovered":43707732,"num_keys_recovered":148,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16281600,"data_stored":109935769,"data_compressed":7469756,"data_compressed_allocated":8933376,"data_compressed_original":103586230,"omap_allocated":18388,"internal_metadata":0},"log_size":15533,"ondisk_log_size":15533,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312056,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94059784,"statfs":{"total":96636764160,"available":96317218816,"internally_reserved":0,"allocated":2269184,"data_stored":4428227,"data_compressed":232116,"data_compressed_allocated":352256,"data_compressed_original":3295422,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.83199999999999996,"15min":0.85199999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.4179999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.2809999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.83799999999999997,"15min":0.85199999999999998},"min":{"1min":0.51600000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0760000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.76200000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.82599999999999996,"15min":0.82399999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.8809999999999998,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":0.876},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.96899999999999997,"15min":0.97599999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.9169999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":1.1419999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.871,"15min":0.89200000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0740000000000001,"5min":1.476,"15min":1.476},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.014,"15min":1.052},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.226,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":0.99099999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.0449999999999999,"15min":1.0820000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0960000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.995,"15min":1.002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":3.0209999999999999,"5min":3.0209999999999999,"15min":3.0209999999999999},"last":0.65200000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.96299999999999997,"15min":0.98699999999999999},"min":{"1min":0.55900000000000005,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1120000000000001,"5min":1.825,"15min":1.825},"last":1.335},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.038,"15min":1.0620000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1910000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.1890000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.89400000000000002,"15min":0.90800000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.3420000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.121,"15min":1.151},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.347,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":1.2230000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.04},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.1040000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509473,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314992,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94056848,"statfs":{"total":96636764160,"available":96314212352,"internally_reserved":0,"allocated":5279744,"data_stored":31880886,"data_compressed":2003397,"data_compressed_allocated":2449408,"data_compressed_original":29990734,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.77900000000000003,"15min":0.78700000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.151,"5min":1.173,"15min":1.173},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.77500000000000002,"15min":0.79900000000000004},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.91400000000000003,"5min":1.927,"15min":1.927},"last":1.0169999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.76500000000000001,"15min":0.77600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.97999999999999998,"5min":1.109,"15min":1.109},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.89400000000000002,"15min":0.91400000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.2669999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.80600000000000005,"15min":0.81599999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.91100000000000003,"15min":0.93899999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0649999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.80000000000000004}]},{"osd":3,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.82199999999999995,"15min":0.83999999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.1020000000000001,"5min":1.236,"15min":1.236},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.92800000000000005,"15min":0.94799999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.107,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.95699999999999996}]},{"osd":5,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.93899999999999995,"15min":0.96699999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.4059999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.079}]},{"osd":6,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.94599999999999995,"15min":0.97999999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.022,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.876},{"interface":"front","average":{"1min":0.85399999999999998,"5min":1.0569999999999999,"15min":1.091},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3460000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.98599999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.125},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.05}]}]},{"osd":3,"up_from":15,"seq":64424509474,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309872,"kb_used_data":4136,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061968,"statfs":{"total":96636764160,"available":96319455232,"internally_reserved":0,"allocated":4235264,"data_stored":15334103,"data_compressed":1147394,"data_compressed_allocated":1179648,"data_compressed_original":13328342,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82899999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.65600000000000003,"15min":0.67400000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":1.03,"15min":1.03},"last":0.58599999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65300000000000002,"15min":0.66400000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.97799999999999998,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.75800000000000001,"15min":0.76800000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.753,"15min":0.76800000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.92600000000000005,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.871,"15min":0.88600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.0109999999999999,"15min":0.98499999999999999},"min":{"1min":0.57799999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":4.9119999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.103,"5min":0.98299999999999998,"15min":0.96299999999999997},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":4.9489999999999998,"5min":4.9489999999999998,"15min":4.9489999999999998},"last":1.0289999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":0.98399999999999999,"15min":0.95299999999999996},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":5.3449999999999998,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0860000000000001,"15min":1.091},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.397,"5min":1.45,"15min":1.45},"last":1.0069999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.048,"15min":1.0569999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0029999999999999,"15min":1.0069999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.3069999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.89800000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.96799999999999997}]}]},{"osd":1,"up_from":15,"seq":64424509474,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301396,"kb_used_data":4028,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070444,"statfs":{"total":96636764160,"available":96328134656,"internally_reserved":0,"allocated":4124672,"data_stored":20659750,"data_compressed":1189566,"data_compressed_allocated":1527808,"data_compressed_original":19024150,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.58299999999999996,"15min":0.59099999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.94599999999999995,"5min":1.004,"15min":1.004},"last":0.81499999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.80000000000000004,"5min":0.872,"15min":0.872},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65900000000000003,"15min":0.66300000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.86099999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.69199999999999995,"15min":0.70599999999999996},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92900000000000005,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":1.034},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71599999999999997,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0449999999999999,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.91300000000000003}]},{"osd":4,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82499999999999996,"15min":0.82899999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0449999999999999,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.95199999999999996,"15min":0.96199999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.323,"5min":1.419,"15min":1.419},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.90100000000000002,"15min":0.91700000000000004},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.073,"5min":1.506,"15min":1.506},"last":1.097},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.91600000000000004,"15min":0.92800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1120000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":1.079}]},{"osd":6,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.97399999999999998,"15min":0.98599999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.1879999999999999,"5min":1.569,"15min":1.569},"last":1.137},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.996,"15min":1.012},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.171,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.121}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.89100000000000001}]}]},{"osd":7,"up_from":107,"seq":459561500677,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307568,"kb_used_data":3996,"kb_used_omap":50,"kb_used_meta":303501,"kb_avail":94064272,"statfs":{"total":96636764160,"available":96321814528,"internally_reserved":0,"allocated":4091904,"data_stored":19045050,"data_compressed":1376524,"data_compressed_allocated":1474560,"data_compressed_original":17383424,"omap_allocated":51722,"internal_metadata":310785526},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]}]},{"osd":5,"up_from":14,"seq":60129542177,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300248,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94071592,"statfs":{"total":96636764160,"available":96329310208,"internally_reserved":0,"allocated":2961408,"data_stored":4969724,"data_compressed":398446,"data_compressed_allocated":356352,"data_compressed_original":3301376,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.749,"15min":0.75600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.117,"5min":0.97299999999999998,"15min":0.94899999999999995},"min":{"1min":0.40000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":9.5120000000000005,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":0.52300000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.78500000000000003,"15min":0.77300000000000002},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86899999999999999,"15min":0.86499999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.94799999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.85299999999999998,"15min":0.86799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.879,"5min":0.94299999999999995,"15min":0.95299999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93100000000000005}]},{"osd":3,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.97599999999999998,"15min":0.95799999999999996},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.0209999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.871,"15min":0.88700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1220000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.874,"5min":1.002,"15min":1.0229999999999999},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3009999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.89600000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.96599999999999997,"15min":0.97799999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.329,"5min":1.637,"15min":1.637},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0529999999999999,"15min":1.0720000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.1910000000000001,"5min":1.661,"15min":1.661},"last":0.82499999999999996}]},{"osd":7,"last update":"Mon Sep 16 13:17:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.0509999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542176,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308196,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063644,"statfs":{"total":96636764160,"available":96321171456,"internally_reserved":0,"allocated":2576384,"data_stored":10711106,"data_compressed":751880,"data_compressed_allocated":774144,"data_compressed_original":9736192,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.59099999999999997,"15min":0.60099999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.81699999999999995,"5min":1.159,"15min":1.159},"last":0.755},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.69899999999999995,"15min":0.71899999999999997},"min":{"1min":0.29399999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.86899999999999999,"5min":1.506,"15min":1.506},"last":1.0649999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.77800000000000002,"15min":0.78700000000000003},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2570000000000001,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.73199999999999998,"15min":0.72399999999999998},"min":{"1min":0.40300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.298,"5min":3.298,"15min":3.298},"last":0.71199999999999997}]},{"osd":3,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.80500000000000005,"15min":0.81000000000000005},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.89800000000000002,"15min":0.92500000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1830000000000001,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.83099999999999996}]},{"osd":4,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.98699999999999999,"15min":1.016},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.127,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.99099999999999999,"15min":0.996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.514,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.88700000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.0660000000000001,"15min":1.0840000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.415,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.085,"15min":1.105},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.3,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.2030000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.05,"15min":1.0680000000000001},"min":{"1min":0.58399999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.605,"5min":1.857,"15min":1.857},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.1459999999999999,"15min":1.1240000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.0090000000000003,"5min":7.0090000000000003,"15min":7.0090000000000003},"last":1.2569999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.038},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":1.1499999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300556,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94071284,"statfs":{"total":96636764160,"available":96328994816,"internally_reserved":0,"allocated":3215360,"data_stored":12377321,"data_compressed":758885,"data_compressed_allocated":1114112,"data_compressed_original":11180222,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.66000000000000003,"15min":0.66800000000000004},"min":{"1min":0.314,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.90300000000000002,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.71999999999999997,"15min":0.71799999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.98699999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.71399999999999997},"min":{"1min":0.39800000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.7350000000000001,"5min":1.74,"15min":1.74},"last":0.69199999999999995}]},{"osd":3,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.77900000000000003,"15min":0.78100000000000003},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.216,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.95399999999999996}]},{"osd":4,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.90900000000000003,"15min":0.91900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.673,"5min":1.871,"15min":1.871},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.93999999999999995,"15min":0.94299999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.877}]},{"osd":5,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.006,"15min":1},"min":{"1min":0.67800000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.2230000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.95599999999999996,"15min":0.95499999999999996},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":1.153}]},{"osd":6,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.03,"15min":1.036},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.4379999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.282},{"interface":"front","average":{"1min":1.089,"5min":1.036,"15min":1.0269999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.0780000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.881,"5min":1.881,"15min":1.881},"last":1.343},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.2070000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":11070587,"data_compressed":724337,"data_compressed_allocated":1069056,"data_compressed_original":10590398,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2596864,"data_stored":19812277,"data_compressed":1183370,"data_compressed_allocated":1519616,"data_compressed_original":18893078,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":884736,"data_stored":7860624,"data_compressed":463086,"data_compressed_allocated":610304,"data_compressed_original":7639040,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2707456,"data_stored":14611450,"data_compressed":1147394,"data_compressed_allocated":1179648,"data_compressed_original":13328342,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":30902330,"data_compressed":1977403,"data_compressed_allocated":2416640,"data_compressed_original":29728590,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1441792,"data_stored":4228304,"data_compressed":398446,"data_compressed_allocated":356352,"data_compressed_original":3301376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":745472,"data_stored":3705612,"data_compressed":232116,"data_compressed_allocated":352256,"data_compressed_original":3295422,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2523136,"data_stored":17744585,"data_compressed":1343604,"data_compressed_allocated":1429504,"data_compressed_original":16809984,"omap_allocated":18388,"internal_metadata":0}]}} 2024-09-16T13:19:06.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:19:06.373 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:06.357+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:06.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:19:06.464 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:06.459+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:06.479 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:06.479 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:06.479 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:06.479 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:06.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:19:06.564 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:06.557+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:06.588 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:06.588 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:06.588 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:06.588 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:06.588 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:06.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:19:06.598 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:19:06.619 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:19:06.636 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:19:06.655 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:19:06.665 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:06.657+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:06.678 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:19:06.693 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:06.693 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:06.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:19:06.706 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:19:06.730 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:06.731 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:19:06.731 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:19:06.732 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:19:06.732 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:19:06.732 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:19:06.732 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:19:06.732 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:19:06.732 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:19:06.732 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:19:06.733 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:19:06.744 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-16T13:19:06.015156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":244198106,"num_objects":690,"num_object_clones":146,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":331,"num_read":140805,"num_read_kb":678295,"num_write":18671,"num_write_kb":297182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":865,"num_bytes_recovered":44298100,"num_keys_recovered":148,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15608,"ondisk_log_size":15608,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2454884,"kb_used_data":28080,"kb_used_omap":62,"kb_used_meta":2426497,"kb_avail":752519836,"statfs":{"total":773094113280,"available":770580312064,"internally_reserved":0,"allocated":28753920,"data_stored":119406167,"data_compressed":7858208,"data_compressed_allocated":9228288,"data_compressed_original":107239862,"omap_allocated":63650,"internal_metadata":2484733790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-174328235,"num_objects":-142,"num_object_clones":-22,"num_object_copies":-284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-142,"num_whiteouts":-19,"num_read":1391,"num_read_kb":1191,"num_write":270,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030411"},"pg_stats":[{"pgid":"3.1f","version":"127'338","reported_seq":1780,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.984517+0000","last_change":"2024-09-16T13:19:02.984823+0000","last_active":"2024-09-16T13:19:02.984517+0000","last_peered":"2024-09-16T13:19:02.984517+0000","last_clean":"2024-09-16T13:19:02.984517+0000","last_became_active":"2024-09-16T13:18:03.119735+0000","last_became_peered":"2024-09-16T13:18:03.119735+0000","last_unstale":"2024-09-16T13:19:02.984517+0000","last_undegraded":"2024-09-16T13:19:02.984517+0000","last_fullsized":"2024-09-16T13:19:02.984517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:02.061478+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0054926480000000001,"stat_sum":{"num_bytes":11337927,"num_objects":16,"num_object_clones":3,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":8,"num_read":2043,"num_read_kb":15887,"num_write":340,"num_write_kb":5580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":922127,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.1e","version":"126'768","reported_seq":2408,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.001925+0000","last_change":"2024-09-16T13:19:04.002399+0000","last_active":"2024-09-16T13:19:04.001925+0000","last_peered":"2024-09-16T13:19:04.001925+0000","last_clean":"2024-09-16T13:19:04.001925+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:19:04.001925+0000","last_undegraded":"2024-09-16T13:19:04.001925+0000","last_fullsized":"2024-09-16T13:19:04.001925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":768,"log_dups_size":0,"ondisk_log_size":768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:50.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001766447,"stat_sum":{"num_bytes":4454089,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":9,"num_read":5463,"num_read_kb":7000,"num_write":964,"num_write_kb":6235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.1d","version":"127'424","reported_seq":2559,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.178506+0000","last_change":"2024-09-16T13:19:02.979901+0000","last_active":"2024-09-16T13:19:03.178506+0000","last_peered":"2024-09-16T13:19:03.178506+0000","last_clean":"2024-09-16T13:19:03.178506+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:19:03.178506+0000","last_undegraded":"2024-09-16T13:19:03.178506+0000","last_fullsized":"2024-09-16T13:19:03.178506+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:14.387827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000638585,"stat_sum":{"num_bytes":2584403,"num_objects":22,"num_object_clones":4,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":11,"num_read":4763,"num_read_kb":13863,"num_write":438,"num_write_kb":6042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.1c","version":"128'421","reported_seq":2583,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.158121+0000","last_change":"2024-09-16T13:19:04.021435+0000","last_active":"2024-09-16T13:19:04.158121+0000","last_peered":"2024-09-16T13:19:04.158121+0000","last_clean":"2024-09-16T13:19:04.158121+0000","last_became_active":"2024-09-16T13:17:10.061217+0000","last_became_peered":"2024-09-16T13:17:10.061217+0000","last_unstale":"2024-09-16T13:19:04.158121+0000","last_undegraded":"2024-09-16T13:19:04.158121+0000","last_fullsized":"2024-09-16T13:19:04.158121+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:10.571278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072099600000000001,"stat_sum":{"num_bytes":3608797,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":9,"num_read":2940,"num_read_kb":25156,"num_write":527,"num_write_kb":12813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.1b","version":"127'628","reported_seq":5473,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.000656+0000","last_change":"2024-09-16T13:19:04.001325+0000","last_active":"2024-09-16T13:19:04.000656+0000","last_peered":"2024-09-16T13:19:04.000656+0000","last_clean":"2024-09-16T13:19:04.000656+0000","last_became_active":"2024-09-16T13:18:44.442336+0000","last_became_peered":"2024-09-16T13:18:44.442336+0000","last_unstale":"2024-09-16T13:19:04.000656+0000","last_undegraded":"2024-09-16T13:19:04.000656+0000","last_fullsized":"2024-09-16T13:19:04.000656+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:46:34.313429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001845232,"stat_sum":{"num_bytes":3148182,"num_objects":31,"num_object_clones":7,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":15,"num_read":6522,"num_read_kb":28075,"num_write":890,"num_write_kb":17338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2924073,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.1a","version":"128'636","reported_seq":7784,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.180571+0000","last_change":"2024-09-16T13:19:04.000062+0000","last_active":"2024-09-16T13:19:04.180571+0000","last_peered":"2024-09-16T13:19:04.180571+0000","last_clean":"2024-09-16T13:19:04.180571+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:19:04.180571+0000","last_undegraded":"2024-09-16T13:19:04.180571+0000","last_fullsized":"2024-09-16T13:19:04.180571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:31.071219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012938120000000001,"stat_sum":{"num_bytes":14964474,"num_objects":20,"num_object_clones":5,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":10,"num_read":9157,"num_read_kb":54961,"num_write":840,"num_write_kb":18791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.19","version":"128'589","reported_seq":5404,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.009499+0000","last_change":"2024-09-16T13:19:04.009747+0000","last_active":"2024-09-16T13:19:04.009499+0000","last_peered":"2024-09-16T13:19:04.009499+0000","last_clean":"2024-09-16T13:19:04.009499+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:19:04.009499+0000","last_undegraded":"2024-09-16T13:19:04.009499+0000","last_fullsized":"2024-09-16T13:19:04.009499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:42:30.327300+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0072239230000000001,"stat_sum":{"num_bytes":5689078,"num_objects":25,"num_object_clones":6,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":12,"num_read":7637,"num_read_kb":32040,"num_write":805,"num_write_kb":13589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.18","version":"128'414","reported_seq":2317,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.006692+0000","last_change":"2024-09-16T13:19:04.007967+0000","last_active":"2024-09-16T13:19:04.006692+0000","last_peered":"2024-09-16T13:19:04.006692+0000","last_clean":"2024-09-16T13:19:04.006692+0000","last_became_active":"2024-09-16T13:17:02.962736+0000","last_became_peered":"2024-09-16T13:17:02.962736+0000","last_unstale":"2024-09-16T13:19:04.006692+0000","last_undegraded":"2024-09-16T13:19:04.006692+0000","last_fullsized":"2024-09-16T13:19:04.006692+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:27:05.338093+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0089036640000000004,"stat_sum":{"num_bytes":2699245,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":6,"num_read":2801,"num_read_kb":8702,"num_write":535,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.17","version":"126'523","reported_seq":2175,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.965129+0000","last_change":"2024-09-16T13:19:02.965403+0000","last_active":"2024-09-16T13:19:02.965129+0000","last_peered":"2024-09-16T13:19:02.965129+0000","last_clean":"2024-09-16T13:19:02.965129+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:19:02.965129+0000","last_undegraded":"2024-09-16T13:19:02.965129+0000","last_fullsized":"2024-09-16T13:19:02.965129+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:08:57.225261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473364,"stat_sum":{"num_bytes":4912227,"num_objects":24,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":11,"num_read":3492,"num_read_kb":13647,"num_write":563,"num_write_kb":7797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.16","version":"126'398","reported_seq":3007,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.980531+0000","last_change":"2024-09-16T13:19:02.981018+0000","last_active":"2024-09-16T13:19:02.980531+0000","last_peered":"2024-09-16T13:19:02.980531+0000","last_clean":"2024-09-16T13:19:02.980531+0000","last_became_active":"2024-09-16T13:18:44.441740+0000","last_became_peered":"2024-09-16T13:18:44.441740+0000","last_unstale":"2024-09-16T13:19:02.980531+0000","last_undegraded":"2024-09-16T13:19:02.980531+0000","last_fullsized":"2024-09-16T13:19:02.980531+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:20.994894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076613699999999996,"stat_sum":{"num_bytes":5597317,"num_objects":15,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":7,"num_read":5473,"num_read_kb":21309,"num_write":353,"num_write_kb":5655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1185011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.15","version":"126'212","reported_seq":724,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.124662+0000","last_change":"2024-09-16T13:19:02.119706+0000","last_active":"2024-09-16T13:19:02.124662+0000","last_peered":"2024-09-16T13:19:02.124662+0000","last_clean":"2024-09-16T13:19:02.124662+0000","last_became_active":"2024-09-16T13:18:44.435407+0000","last_became_peered":"2024-09-16T13:18:44.435407+0000","last_unstale":"2024-09-16T13:19:02.124662+0000","last_undegraded":"2024-09-16T13:19:02.124662+0000","last_fullsized":"2024-09-16T13:19:02.124662+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:11.546854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031633669999999998,"stat_sum":{"num_bytes":8133452,"num_objects":23,"num_object_clones":7,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":13,"num_read":3249,"num_read_kb":15295,"num_write":187,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1086312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]},{"pgid":"3.14","version":"126'296","reported_seq":4974,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.999237+0000","last_change":"2024-09-16T13:19:04.000058+0000","last_active":"2024-09-16T13:19:03.999237+0000","last_peered":"2024-09-16T13:19:03.999237+0000","last_clean":"2024-09-16T13:19:03.999237+0000","last_became_active":"2024-09-16T13:18:47.462532+0000","last_became_peered":"2024-09-16T13:18:47.462532+0000","last_unstale":"2024-09-16T13:19:03.999237+0000","last_undegraded":"2024-09-16T13:19:03.999237+0000","last_fullsized":"2024-09-16T13:19:03.999237+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:16.599784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288242,"stat_sum":{"num_bytes":2990273,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":5,"num_read":4733,"num_read_kb":25384,"num_write":343,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1053570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.7","version":"0'0","reported_seq":137,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.997965+0000","last_change":"2024-09-16T13:18:44.438193+0000","last_active":"2024-09-16T13:19:02.997965+0000","last_peered":"2024-09-16T13:19:02.997965+0000","last_clean":"2024-09-16T13:19:02.997965+0000","last_became_active":"2024-09-16T13:18:44.437778+0000","last_became_peered":"2024-09-16T13:18:44.437778+0000","last_unstale":"2024-09-16T13:19:02.997965+0000","last_undegraded":"2024-09-16T13:19:02.997965+0000","last_fullsized":"2024-09-16T13:19:02.997965+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:34.121552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"127'665","reported_seq":5541,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.965762+0000","last_change":"2024-09-16T13:19:02.965762+0000","last_active":"2024-09-16T13:19:02.965762+0000","last_peered":"2024-09-16T13:19:02.965762+0000","last_clean":"2024-09-16T13:19:02.965762+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:19:02.965762+0000","last_undegraded":"2024-09-16T13:19:02.965762+0000","last_fullsized":"2024-09-16T13:19:02.965762+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:52:05.144800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001692762,"stat_sum":{"num_bytes":5398686,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":11,"num_read":6650,"num_read_kb":24623,"num_write":934,"num_write_kb":11031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"2.6","version":"0'0","reported_seq":122,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.984554+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:19:02.984554+0000","last_peered":"2024-09-16T13:19:02.984554+0000","last_clean":"2024-09-16T13:19:02.984554+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:19:02.984554+0000","last_undegraded":"2024-09-16T13:19:02.984554+0000","last_fullsized":"2024-09-16T13:19:02.984554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"128'290","reported_seq":1378,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.007430+0000","last_change":"2024-09-16T13:19:04.008193+0000","last_active":"2024-09-16T13:19:04.007430+0000","last_peered":"2024-09-16T13:19:04.007430+0000","last_clean":"2024-09-16T13:19:04.007430+0000","last_became_active":"2024-09-16T13:18:47.463039+0000","last_became_peered":"2024-09-16T13:18:47.463039+0000","last_unstale":"2024-09-16T13:19:04.007430+0000","last_undegraded":"2024-09-16T13:19:04.007430+0000","last_fullsized":"2024-09-16T13:19:04.007430+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:05:53.068034+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0073940910000000002,"stat_sum":{"num_bytes":5070848,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":13,"num_read":1655,"num_read_kb":8730,"num_write":267,"num_write_kb":5747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1061466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.5","version":"0'0","reported_seq":102,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T13:19:00.994174+0000","last_change":"2024-09-16T13:18:44.435471+0000","last_active":"2024-09-16T13:19:00.994174+0000","last_peered":"2024-09-16T13:19:00.994174+0000","last_clean":"2024-09-16T13:19:00.994174+0000","last_became_active":"2024-09-16T13:18:44.434851+0000","last_became_peered":"2024-09-16T13:18:44.434851+0000","last_unstale":"2024-09-16T13:19:00.994174+0000","last_undegraded":"2024-09-16T13:19:00.994174+0000","last_fullsized":"2024-09-16T13:19:00.994174+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:12.980758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"127'360","reported_seq":4594,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.002823+0000","last_change":"2024-09-16T13:19:04.003241+0000","last_active":"2024-09-16T13:19:04.002823+0000","last_peered":"2024-09-16T13:19:04.002823+0000","last_clean":"2024-09-16T13:19:04.002823+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:19:04.002823+0000","last_undegraded":"2024-09-16T13:19:04.002823+0000","last_fullsized":"2024-09-16T13:19:04.002823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:30.996264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059763199999999996,"stat_sum":{"num_bytes":1355248,"num_objects":16,"num_object_clones":2,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":6,"num_read":4693,"num_read_kb":26876,"num_write":414,"num_write_kb":8111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.4","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.984788+0000","last_change":"2024-09-16T13:17:42.960492+0000","last_active":"2024-09-16T13:19:02.984788+0000","last_peered":"2024-09-16T13:19:02.984788+0000","last_clean":"2024-09-16T13:19:02.984788+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:19:02.984788+0000","last_undegraded":"2024-09-16T13:19:02.984788+0000","last_fullsized":"2024-09-16T13:19:02.984788+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:35:45.152557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"126'511","reported_seq":3717,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.979982+0000","last_change":"2024-09-16T13:19:02.980468+0000","last_active":"2024-09-16T13:19:02.979982+0000","last_peered":"2024-09-16T13:19:02.979982+0000","last_clean":"2024-09-16T13:19:02.979982+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:19:02.979982+0000","last_undegraded":"2024-09-16T13:19:02.979982+0000","last_fullsized":"2024-09-16T13:19:02.979982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:34.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00110391,"stat_sum":{"num_bytes":12552397,"num_objects":33,"num_object_clones":6,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":17,"num_read":5910,"num_read_kb":14107,"num_write":521,"num_write_kb":4259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.3","version":"128'435","reported_seq":2249,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.207566+0000","last_change":"2024-09-16T13:19:04.000634+0000","last_active":"2024-09-16T13:19:04.207566+0000","last_peered":"2024-09-16T13:19:04.207566+0000","last_clean":"2024-09-16T13:19:04.207566+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:19:04.207566+0000","last_undegraded":"2024-09-16T13:19:04.207566+0000","last_fullsized":"2024-09-16T13:19:04.207566+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:44:16.402879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001574317,"stat_sum":{"num_bytes":2465952,"num_objects":19,"num_object_clones":4,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":11,"num_read":2497,"num_read_kb":21118,"num_write":430,"num_write_kb":8994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.2","version":"21'2","reported_seq":123,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.115292+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:19:02.115292+0000","last_peered":"2024-09-16T13:19:02.115292+0000","last_clean":"2024-09-16T13:19:02.115292+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:19:02.115292+0000","last_undegraded":"2024-09-16T13:19:02.115292+0000","last_fullsized":"2024-09-16T13:19:02.115292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"127'408","reported_seq":1687,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.003324+0000","last_change":"2024-09-16T13:19:04.004813+0000","last_active":"2024-09-16T13:19:04.003324+0000","last_peered":"2024-09-16T13:19:04.003324+0000","last_clean":"2024-09-16T13:19:04.003324+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:19:04.003324+0000","last_undegraded":"2024-09-16T13:19:04.003324+0000","last_fullsized":"2024-09-16T13:19:04.003324+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:29.090161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001088489,"stat_sum":{"num_bytes":6388051,"num_objects":24,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":13,"num_read":3120,"num_read_kb":11028,"num_write":469,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.1","version":"0'0","reported_seq":121,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.133127+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:19:02.133127+0000","last_peered":"2024-09-16T13:19:02.133127+0000","last_clean":"2024-09-16T13:19:02.133127+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:19:02.133127+0000","last_undegraded":"2024-09-16T13:19:02.133127+0000","last_fullsized":"2024-09-16T13:19:02.133127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"126'678","reported_seq":2095,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.965061+0000","last_change":"2024-09-16T13:19:02.965479+0000","last_active":"2024-09-16T13:19:02.965061+0000","last_peered":"2024-09-16T13:19:02.965061+0000","last_clean":"2024-09-16T13:19:02.965061+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:19:02.965061+0000","last_undegraded":"2024-09-16T13:19:02.965061+0000","last_fullsized":"2024-09-16T13:19:02.965061+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:03.357310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015907569999999999,"stat_sum":{"num_bytes":13625373,"num_objects":28,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":16,"num_read":3850,"num_read_kb":31161,"num_write":947,"num_write_kb":16370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"2.0","version":"0'0","reported_seq":89,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T13:19:00.994124+0000","last_change":"2024-09-16T13:18:44.435672+0000","last_active":"2024-09-16T13:19:00.994124+0000","last_peered":"2024-09-16T13:19:00.994124+0000","last_clean":"2024-09-16T13:19:00.994124+0000","last_became_active":"2024-09-16T13:18:44.435188+0000","last_became_peered":"2024-09-16T13:18:44.435188+0000","last_unstale":"2024-09-16T13:19:00.994124+0000","last_undegraded":"2024-09-16T13:19:00.994124+0000","last_fullsized":"2024-09-16T13:19:00.994124+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:35.400164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"126'453","reported_seq":2034,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:03.999594+0000","last_change":"2024-09-16T13:19:04.000215+0000","last_active":"2024-09-16T13:19:03.999594+0000","last_peered":"2024-09-16T13:19:03.999594+0000","last_clean":"2024-09-16T13:19:03.999594+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:19:03.999594+0000","last_undegraded":"2024-09-16T13:19:03.999594+0000","last_fullsized":"2024-09-16T13:19:03.999594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:45.154984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085824899999999995,"stat_sum":{"num_bytes":1800829,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":13,"num_read":3321,"num_read_kb":15348,"num_write":485,"num_write_kb":6255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"2.3","version":"19'1","reported_seq":122,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.115273+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:19:02.115273+0000","last_peered":"2024-09-16T13:19:02.115273+0000","last_clean":"2024-09-16T13:19:02.115273+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:19:02.115273+0000","last_undegraded":"2024-09-16T13:19:02.115273+0000","last_fullsized":"2024-09-16T13:19:02.115273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":202,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-16T13:19:00.994294+0000","last_change":"2024-09-16T13:18:44.437802+0000","last_active":"2024-09-16T13:19:00.994294+0000","last_peered":"2024-09-16T13:19:00.994294+0000","last_clean":"2024-09-16T13:19:00.994294+0000","last_became_active":"2024-09-16T13:18:44.437418+0000","last_became_peered":"2024-09-16T13:18:44.437418+0000","last_unstale":"2024-09-16T13:19:00.994294+0000","last_undegraded":"2024-09-16T13:19:00.994294+0000","last_fullsized":"2024-09-16T13:19:00.994294+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:29.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"122'578","reported_seq":1192,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.001395+0000","last_change":"2024-09-16T13:19:04.001798+0000","last_active":"2024-09-16T13:19:04.001395+0000","last_peered":"2024-09-16T13:19:04.001395+0000","last_clean":"2024-09-16T13:19:04.001395+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:19:04.001395+0000","last_undegraded":"2024-09-16T13:19:04.001395+0000","last_fullsized":"2024-09-16T13:19:04.001395+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":578,"log_dups_size":0,"ondisk_log_size":578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:59.515115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022672669999999999,"stat_sum":{"num_bytes":10415520,"num_objects":20,"num_object_clones":3,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":6,"num_read":1983,"num_read_kb":19347,"num_write":765,"num_write_kb":12166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.9","version":"126'477","reported_seq":2812,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.166545+0000","last_change":"2024-09-16T13:19:04.000549+0000","last_active":"2024-09-16T13:19:04.166545+0000","last_peered":"2024-09-16T13:19:04.166545+0000","last_clean":"2024-09-16T13:19:04.166545+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:19:04.166545+0000","last_undegraded":"2024-09-16T13:19:04.166545+0000","last_fullsized":"2024-09-16T13:19:04.166545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:49:10.301550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017042649999999999,"stat_sum":{"num_bytes":16858086,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":14,"num_read":4015,"num_read_kb":31205,"num_write":422,"num_write_kb":12345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.a","version":"128'908","reported_seq":2908,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.118957+0000","last_change":"2024-09-16T13:19:04.021081+0000","last_active":"2024-09-16T13:19:04.118957+0000","last_peered":"2024-09-16T13:19:04.118957+0000","last_clean":"2024-09-16T13:19:04.118957+0000","last_became_active":"2024-09-16T13:16:52.299458+0000","last_became_peered":"2024-09-16T13:16:52.299458+0000","last_unstale":"2024-09-16T13:19:04.118957+0000","last_undegraded":"2024-09-16T13:19:04.118957+0000","last_fullsized":"2024-09-16T13:19:04.118957+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":908,"log_dups_size":0,"ondisk_log_size":908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:00:34.071306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038801599999999998,"stat_sum":{"num_bytes":6322236,"num_objects":22,"num_object_clones":5,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":8,"num_read":4714,"num_read_kb":19938,"num_write":1249,"num_write_kb":12318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.b","version":"128'1072","reported_seq":4160,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.209036+0000","last_change":"2024-09-16T13:19:04.001302+0000","last_active":"2024-09-16T13:19:04.209036+0000","last_peered":"2024-09-16T13:19:04.209036+0000","last_clean":"2024-09-16T13:19:04.209036+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:19:04.209036+0000","last_undegraded":"2024-09-16T13:19:04.209036+0000","last_fullsized":"2024-09-16T13:19:04.209036+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:20.684083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018441429999999999,"stat_sum":{"num_bytes":1850738,"num_objects":19,"num_object_clones":2,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":6,"num_read":9495,"num_read_kb":27050,"num_write":1643,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.c","version":"124'286","reported_seq":1263,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.979265+0000","last_change":"2024-09-16T13:19:02.979820+0000","last_active":"2024-09-16T13:19:02.979265+0000","last_peered":"2024-09-16T13:19:02.979265+0000","last_clean":"2024-09-16T13:19:02.979265+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:19:02.979265+0000","last_undegraded":"2024-09-16T13:19:02.979265+0000","last_fullsized":"2024-09-16T13:19:02.979265+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:39.937083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080311499999999997,"stat_sum":{"num_bytes":9267082,"num_objects":17,"num_object_clones":3,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":1274,"num_read_kb":14888,"num_write":268,"num_write_kb":7028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.d","version":"126'359","reported_seq":2304,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.263694+0000","last_change":"2024-09-16T13:19:02.120551+0000","last_active":"2024-09-16T13:19:02.263694+0000","last_peered":"2024-09-16T13:19:02.263694+0000","last_clean":"2024-09-16T13:19:02.263694+0000","last_became_active":"2024-09-16T13:18:47.468001+0000","last_became_peered":"2024-09-16T13:18:47.468001+0000","last_unstale":"2024-09-16T13:19:02.263694+0000","last_undegraded":"2024-09-16T13:19:02.263694+0000","last_fullsized":"2024-09-16T13:19:02.263694+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:11:32.886561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002450308,"stat_sum":{"num_bytes":20989652,"num_objects":25,"num_object_clones":3,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":14,"num_read":2646,"num_read_kb":18369,"num_write":325,"num_write_kb":14195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]},{"pgid":"3.e","version":"126'502","reported_seq":6995,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.120711+0000","last_change":"2024-09-16T13:19:02.118898+0000","last_active":"2024-09-16T13:19:02.120711+0000","last_peered":"2024-09-16T13:19:02.120711+0000","last_clean":"2024-09-16T13:19:02.120711+0000","last_became_active":"2024-09-16T13:18:47.467028+0000","last_became_peered":"2024-09-16T13:18:47.467028+0000","last_unstale":"2024-09-16T13:19:02.120711+0000","last_undegraded":"2024-09-16T13:19:02.120711+0000","last_fullsized":"2024-09-16T13:19:02.120711+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:35.852378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023549669999999999,"stat_sum":{"num_bytes":2653579,"num_objects":15,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":9,"num_read":8102,"num_read_kb":29094,"num_write":566,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1715710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]},{"pgid":"3.f","version":"125'268","reported_seq":867,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.120097+0000","last_change":"2024-09-16T13:19:02.120441+0000","last_active":"2024-09-16T13:19:02.120097+0000","last_peered":"2024-09-16T13:19:02.120097+0000","last_clean":"2024-09-16T13:19:02.120097+0000","last_became_active":"2024-09-16T13:18:44.444025+0000","last_became_peered":"2024-09-16T13:18:44.444025+0000","last_unstale":"2024-09-16T13:19:02.120097+0000","last_undegraded":"2024-09-16T13:19:02.120097+0000","last_fullsized":"2024-09-16T13:19:02.120097+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:25:03.521529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001780516,"stat_sum":{"num_bytes":39638841,"num_objects":39,"num_object_clones":8,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":16,"num_read":658,"num_read_kb":10363,"num_write":256,"num_write_kb":7710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":63719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]},{"pgid":"3.10","version":"126'245","reported_seq":1304,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-16T13:19:04.020803+0000","last_change":"2024-09-16T13:19:04.021059+0000","last_active":"2024-09-16T13:19:04.020803+0000","last_peered":"2024-09-16T13:19:04.020803+0000","last_clean":"2024-09-16T13:19:04.020803+0000","last_became_active":"2024-09-16T13:17:42.961695+0000","last_became_peered":"2024-09-16T13:17:42.961695+0000","last_unstale":"2024-09-16T13:19:04.020803+0000","last_undegraded":"2024-09-16T13:19:04.020803+0000","last_fullsized":"2024-09-16T13:19:04.020803+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:35.675244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038742900000000001,"stat_sum":{"num_bytes":9430584,"num_objects":23,"num_object_clones":7,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":10,"num_read":784,"num_read_kb":16305,"num_write":251,"num_write_kb":11082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"8"}]},{"pgid":"3.11","version":"126'545","reported_seq":2371,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.136740+0000","last_change":"2024-09-16T13:19:02.115710+0000","last_active":"2024-09-16T13:19:02.136740+0000","last_peered":"2024-09-16T13:19:02.136740+0000","last_clean":"2024-09-16T13:19:02.136740+0000","last_became_active":"2024-09-16T13:18:47.462561+0000","last_became_peered":"2024-09-16T13:18:47.462561+0000","last_unstale":"2024-09-16T13:19:02.136740+0000","last_undegraded":"2024-09-16T13:19:02.136740+0000","last_fullsized":"2024-09-16T13:19:02.136740+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":545,"log_dups_size":0,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.079652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080268500000000003,"stat_sum":{"num_bytes":3317604,"num_objects":17,"num_object_clones":3,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":9,"num_read":3854,"num_read_kb":23841,"num_write":727,"num_write_kb":11756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]},{"pgid":"3.12","version":"123'390","reported_seq":7415,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.965549+0000","last_change":"2024-09-16T13:19:02.966110+0000","last_active":"2024-09-16T13:19:02.965549+0000","last_peered":"2024-09-16T13:19:02.965549+0000","last_clean":"2024-09-16T13:19:02.965549+0000","last_became_active":"2024-09-16T13:18:47.462412+0000","last_became_peered":"2024-09-16T13:18:47.462412+0000","last_unstale":"2024-09-16T13:19:02.965549+0000","last_undegraded":"2024-09-16T13:19:02.965549+0000","last_fullsized":"2024-09-16T13:19:02.965549+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:50:25.256657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017562820000000001,"stat_sum":{"num_bytes":1572033,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":5,"num_read":8501,"num_read_kb":35036,"num_write":358,"num_write_kb":9220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2494648,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"7"}]},{"pgid":"3.13","version":"126'456","reported_seq":1851,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-16T13:19:02.232677+0000","last_change":"2024-09-16T13:19:02.116504+0000","last_active":"2024-09-16T13:19:02.232677+0000","last_peered":"2024-09-16T13:19:02.232677+0000","last_clean":"2024-09-16T13:19:02.232677+0000","last_became_active":"2024-09-16T13:18:44.443832+0000","last_became_peered":"2024-09-16T13:18:44.443832+0000","last_unstale":"2024-09-16T13:19:02.232677+0000","last_undegraded":"2024-09-16T13:19:02.232677+0000","last_fullsized":"2024-09-16T13:19:02.232677+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:45.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080056599999999995,"stat_sum":{"num_bytes":2514916,"num_objects":26,"num_object_clones":5,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":13,"num_read":4744,"num_read_kb":18494,"num_write":486,"num_write_kb":7694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1653183,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"a"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":243607719,"num_objects":686,"num_object_clones":146,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":331,"num_read":140739,"num_read_kb":678240,"num_write":18568,"num_write_kb":295552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":863,"num_bytes_recovered":43707732,"num_keys_recovered":148,"num_objects_omap":15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16281600,"data_stored":109935769,"data_compressed":7469756,"data_compressed_allocated":8933376,"data_compressed_original":103586230,"omap_allocated":18388,"internal_metadata":0},"log_size":15533,"ondisk_log_size":15533,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312056,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94059784,"statfs":{"total":96636764160,"available":96317218816,"internally_reserved":0,"allocated":2269184,"data_stored":4428227,"data_compressed":232116,"data_compressed_allocated":352256,"data_compressed_original":3295422,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.83199999999999996,"15min":0.85199999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.4179999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.2809999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.83799999999999997,"15min":0.85199999999999998},"min":{"1min":0.51600000000000001,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0760000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.76200000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.82599999999999996,"15min":0.82399999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.8809999999999998,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":0.876},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.96899999999999997,"15min":0.97599999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.9169999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":1.1419999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.871,"15min":0.89200000000000002},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0740000000000001,"5min":1.476,"15min":1.476},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.014,"15min":1.052},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.226,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":0.99099999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":1.0449999999999999,"15min":1.0820000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0960000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.995,"15min":1.002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":3.0209999999999999,"5min":3.0209999999999999,"15min":3.0209999999999999},"last":0.65200000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.96299999999999997,"15min":0.98699999999999999},"min":{"1min":0.55900000000000005,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1120000000000001,"5min":1.825,"15min":1.825},"last":1.335},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.038,"15min":1.0620000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1910000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.1890000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.89400000000000002,"15min":0.90800000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.3420000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.93700000000000006,"5min":1.121,"15min":1.151},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.347,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":1.2230000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.04},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.1040000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509473,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314992,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94056848,"statfs":{"total":96636764160,"available":96314212352,"internally_reserved":0,"allocated":5279744,"data_stored":31880886,"data_compressed":2003397,"data_compressed_allocated":2449408,"data_compressed_original":29990734,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.77900000000000003,"15min":0.78700000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.151,"5min":1.173,"15min":1.173},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.77500000000000002,"15min":0.79900000000000004},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.91400000000000003,"5min":1.927,"15min":1.927},"last":1.0169999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.76500000000000001,"15min":0.77600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.97999999999999998,"5min":1.109,"15min":1.109},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.89400000000000002,"15min":0.91400000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.2669999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.80600000000000005,"15min":0.81599999999999995},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.91100000000000003,"15min":0.93899999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0649999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.80000000000000004}]},{"osd":3,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.82199999999999995,"15min":0.83999999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.1020000000000001,"5min":1.236,"15min":1.236},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.92800000000000005,"15min":0.94799999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.107,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.95699999999999996}]},{"osd":5,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.93899999999999995,"15min":0.96699999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.4059999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.079}]},{"osd":6,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.94599999999999995,"15min":0.97999999999999998},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.022,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.876},{"interface":"front","average":{"1min":0.85399999999999998,"5min":1.0569999999999999,"15min":1.091},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3460000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.98599999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.125},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.05}]}]},{"osd":3,"up_from":15,"seq":64424509474,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309872,"kb_used_data":4136,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94061968,"statfs":{"total":96636764160,"available":96319455232,"internally_reserved":0,"allocated":4235264,"data_stored":15334103,"data_compressed":1147394,"data_compressed_allocated":1179648,"data_compressed_original":13328342,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82899999999999996,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.65600000000000003,"15min":0.67400000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":1.03,"15min":1.03},"last":0.58599999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65300000000000002,"15min":0.66400000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.97799999999999998,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.75800000000000001,"15min":0.76800000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.753,"15min":0.76800000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.92600000000000005,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.871,"15min":0.88600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.0109999999999999,"15min":0.98499999999999999},"min":{"1min":0.57799999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":4.9119999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.103,"5min":0.98299999999999998,"15min":0.96299999999999997},"min":{"1min":0.65900000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":4.9489999999999998,"5min":4.9489999999999998,"15min":4.9489999999999998},"last":1.0289999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":0.98399999999999999,"15min":0.95299999999999996},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":5.3449999999999998,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0860000000000001,"15min":1.091},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.397,"5min":1.45,"15min":1.45},"last":1.0069999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.048,"15min":1.0569999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.0029999999999999,"15min":1.0069999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.3069999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.89800000000000002}]},{"osd":7,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.96799999999999997}]}]},{"osd":1,"up_from":15,"seq":64424509474,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301396,"kb_used_data":4028,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070444,"statfs":{"total":96636764160,"available":96328134656,"internally_reserved":0,"allocated":4124672,"data_stored":20659750,"data_compressed":1189566,"data_compressed_allocated":1527808,"data_compressed_original":19024150,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.58299999999999996,"15min":0.59099999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.94599999999999995,"5min":1.004,"15min":1.004},"last":0.81499999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57499999999999996,"15min":0.57599999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.80000000000000004,"5min":0.872,"15min":0.872},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65900000000000003,"15min":0.66300000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.86099999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.69199999999999995,"15min":0.70599999999999996},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92900000000000005,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":1.034},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.71599999999999997,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0449999999999999,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.91300000000000003}]},{"osd":4,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82499999999999996,"15min":0.82899999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0449999999999999,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.95199999999999996,"15min":0.96199999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.323,"5min":1.419,"15min":1.419},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.90100000000000002,"15min":0.91700000000000004},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.073,"5min":1.506,"15min":1.506},"last":1.097},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.91600000000000004,"15min":0.92800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1120000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":1.079}]},{"osd":6,"last update":"Mon Sep 16 13:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.97399999999999998,"15min":0.98599999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.1879999999999999,"5min":1.569,"15min":1.569},"last":1.137},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.996,"15min":1.012},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.171,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.121}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.89100000000000001}]}]},{"osd":7,"up_from":107,"seq":459561500677,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307568,"kb_used_data":3996,"kb_used_omap":50,"kb_used_meta":303501,"kb_avail":94064272,"statfs":{"total":96636764160,"available":96321814528,"internally_reserved":0,"allocated":4091904,"data_stored":19045050,"data_compressed":1376524,"data_compressed_allocated":1474560,"data_compressed_original":17383424,"omap_allocated":51722,"internal_metadata":310785526},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]}]},{"osd":5,"up_from":14,"seq":60129542177,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300248,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94071592,"statfs":{"total":96636764160,"available":96329310208,"internally_reserved":0,"allocated":2961408,"data_stored":4969724,"data_compressed":398446,"data_compressed_allocated":356352,"data_compressed_original":3301376,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.749,"15min":0.75600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.117,"5min":0.97299999999999998,"15min":0.94899999999999995},"min":{"1min":0.40000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":9.5120000000000005,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":0.52300000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.78500000000000003,"15min":0.77300000000000002},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86899999999999999,"15min":0.86499999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.94799999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:18:36 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.85299999999999998,"15min":0.86799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.879,"5min":0.94299999999999995,"15min":0.95299999999999996},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.93100000000000005}]},{"osd":3,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.97599999999999998,"15min":0.95799999999999996},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.0209999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.871,"15min":0.88700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1220000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.874,"5min":1.002,"15min":1.0229999999999999},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3009999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.89600000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.96599999999999997,"15min":0.97799999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.329,"5min":1.637,"15min":1.637},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0529999999999999,"15min":1.0720000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.1910000000000001,"5min":1.661,"15min":1.661},"last":0.82499999999999996}]},{"osd":7,"last update":"Mon Sep 16 13:17:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.0509999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542176,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308196,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94063644,"statfs":{"total":96636764160,"available":96321171456,"internally_reserved":0,"allocated":2576384,"data_stored":10711106,"data_compressed":751880,"data_compressed_allocated":774144,"data_compressed_original":9736192,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.59099999999999997,"15min":0.60099999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.81699999999999995,"5min":1.159,"15min":1.159},"last":0.755},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.69899999999999995,"15min":0.71899999999999997},"min":{"1min":0.29399999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.86899999999999999,"5min":1.506,"15min":1.506},"last":1.0649999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.77800000000000002,"15min":0.78700000000000003},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2570000000000001,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.73199999999999998,"15min":0.72399999999999998},"min":{"1min":0.40300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":3.298,"5min":3.298,"15min":3.298},"last":0.71199999999999997}]},{"osd":3,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.80500000000000005,"15min":0.81000000000000005},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.89800000000000002,"15min":0.92500000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1830000000000001,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.83099999999999996}]},{"osd":4,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.98699999999999999,"15min":1.016},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.127,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.99099999999999999,"15min":0.996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.514,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.88700000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.0660000000000001,"15min":1.0840000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.415,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.085,"15min":1.105},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.3,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.2030000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:18:37 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.05,"15min":1.0680000000000001},"min":{"1min":0.58399999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.605,"5min":1.857,"15min":1.857},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.1459999999999999,"15min":1.1240000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.0090000000000003,"5min":7.0090000000000003,"15min":7.0090000000000003},"last":1.2569999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.038},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":1.1499999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300556,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94071284,"statfs":{"total":96636764160,"available":96328994816,"internally_reserved":0,"allocated":3215360,"data_stored":12377321,"data_compressed":758885,"data_compressed_allocated":1114112,"data_compressed_original":11180222,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.66000000000000003,"15min":0.66800000000000004},"min":{"1min":0.314,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.90300000000000002,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.71999999999999997,"15min":0.71799999999999997},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.98699999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.71099999999999997,"15min":0.71399999999999997},"min":{"1min":0.39800000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.7350000000000001,"5min":1.74,"15min":1.74},"last":0.69199999999999995}]},{"osd":3,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.77900000000000003,"15min":0.78100000000000003},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.216,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.95399999999999996}]},{"osd":4,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.90900000000000003,"15min":0.91900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.673,"5min":1.871,"15min":1.871},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.93999999999999995,"15min":0.94299999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.877}]},{"osd":5,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.006,"15min":1},"min":{"1min":0.67800000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.2230000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.95599999999999996,"15min":0.95499999999999996},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":1.153}]},{"osd":6,"last update":"Mon Sep 16 13:18:35 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.03,"15min":1.036},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.4379999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.282},{"interface":"front","average":{"1min":1.089,"5min":1.036,"15min":1.0269999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.0780000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.881,"5min":1.881,"15min":1.881},"last":1.343},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.2070000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":11070587,"data_compressed":724337,"data_compressed_allocated":1069056,"data_compressed_original":10590398,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2596864,"data_stored":19812277,"data_compressed":1183370,"data_compressed_allocated":1519616,"data_compressed_original":18893078,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":884736,"data_stored":7860624,"data_compressed":463086,"data_compressed_allocated":610304,"data_compressed_original":7639040,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2707456,"data_stored":14611450,"data_compressed":1147394,"data_compressed_allocated":1179648,"data_compressed_original":13328342,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":30902330,"data_compressed":1977403,"data_compressed_allocated":2416640,"data_compressed_original":29728590,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1441792,"data_stored":4228304,"data_compressed":398446,"data_compressed_allocated":356352,"data_compressed_original":3301376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":745472,"data_stored":3705612,"data_compressed":232116,"data_compressed_allocated":352256,"data_compressed_original":3295422,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2523136,"data_stored":17744585,"data_compressed":1343604,"data_compressed_allocated":1429504,"data_compressed_original":16809984,"omap_allocated":18388,"internal_metadata":0}]}} 2024-09-16T13:19:06.747 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T13:19:06.747 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:19:06.747 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-16T13:19:06.748 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:19:06.748 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:19:06.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-16T13:19:06.790 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:06.785+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:06.827 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:06.827 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:06.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:19:06.865 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:06.865 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T13:19:06.865 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:06.866 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:06.861+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:06.936 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:06.936 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:06.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:19:06.967 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:06.959+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:07.045 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:07.045 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:07.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:19:07.069 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:07.061+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:07.151 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:07.152 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:07.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:19:07.169 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:07.161+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:07.261 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:07.261 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:07.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:19:07.270 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:07.263+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:07.371 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:07.365+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:07.416 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:07.416 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:07.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:19:07.472 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:07.465+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:07.526 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:07.526 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:07.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:19:07.573 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:07.565+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:07.632 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:07.632 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:07.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:19:07.674 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:07.667+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:07.738 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:07.739 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:07.747 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:19:07.788 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:07.783+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:07.860 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:07.860 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:07.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:19:07.876 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:07.869+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:07.973 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:07.973 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:07.977 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:07.969+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:07.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:19:08.045 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:19:08.090 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:08.085+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:08.165 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:08.240 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:08.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:19:08.246 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:08.237+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:08.343 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:08.337+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:08.348 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:08.348 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:08.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:19:08.444 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:08.437+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:08.465 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:08.465 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:08.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:19:08.545 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:08.537+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:08.620 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:08.620 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:08.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:19:08.646 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:08.639+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:08.733 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:08.733 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:08.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:19:08.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:19:08.747 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:08.741+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:08.836 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:08.837 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:08.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:19:08.847 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:08.841+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:08.871 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:19:08.889 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:19:08.918 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:19:08.938 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:19:08.945 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:19:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:08.945 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 124.932861293, 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:08.946 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:19:08.947 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.948 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 124.93169437100001, 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.951 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.952 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6751 3.0 e36/23 3:01737761:::rbd_data.112d6fb8d387.0000000000000137:head v 36'42, mlcod=36'40)", 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.882350605, 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039737210000000004, 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.954 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058689+0000", 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058691+0000", 2024-09-16T13:19:08.955 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.464e-06 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058709+0000", 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8168999999999998e-05 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058756+0000", 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7015000000000003e-05 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.956 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.058778+0000", 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2164e-05 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059012+0000", 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00023402399999999999 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059066+0000", 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3161999999999997e-05 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062569+0000", 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0035032850000000001 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.062662+0000", 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3437999999999995e-05 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6749 3.9 e36/23 3:942632c1:::rbd_data.112d6fb8d387.000000000000009d:head v 36'29, mlcod=34'27)", 2024-09-16T13:19:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:19:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.88174116499999, 2024-09-16T13:19:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003971161, 2024-09-16T13:19:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.959 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059298+0000", 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059300+0000", 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.216e-06 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059318+0000", 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7557000000000001e-05 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059350+0000", 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1962000000000003e-05 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.961 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059371+0000", 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0733e-05 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059487+0000", 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000116408 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.059517+0000", 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0213e-05 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.962 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063231+0000", 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037142870000000001 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.063269+0000", 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7784999999999998e-05 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6964 3.0 e37/23 3:043ce73a:::rbd_data.112d6fb8d387.0000000000000151:head v 37'47, mlcod=37'45)", 2024-09-16T13:19:08.963 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.38445735099999, 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043786470000000003, 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556582+0000", 2024-09-16T13:19:08.964 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556583+0000", 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5740000000000001e-06 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556598+0000", 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4473e-05 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.965 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556624+0000", 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5902e-05 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556640+0000", 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5787e-05 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.556958+0000", 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00031795599999999999 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.966 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557056+0000", 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.8641000000000006e-05 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560852+0000", 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003795779 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.560961+0000", 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000108535 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.383356535, 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:19:08.969 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.970 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.971 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:19:08.972 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.383216895, 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.974 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.975 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.976 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 116.38271636499999, 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.977 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.978 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.979 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.980 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 111.38754697900001, 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.981 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 101.131133138, 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.986 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.987 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.988 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 100.518570031, 2024-09-16T13:19:08.989 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:19:08.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.993 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 73.680463861000007, 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:08.994 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:19:08.995 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:08.996 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:08.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 73.316336527999994, 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:19:08.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:19:09.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:19:09.001 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:75420 3.4 e88/23 3:2605e3b8:::rbd_object_map.112d9326a5dd.0000000000000026:head v 88'165, mlcod=88'163)", 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 49.437607512, 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055875079999999997, 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.002 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503433+0000", 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503451+0000", 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7640999999999999e-05 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503476+0000", 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5599999999999999e-05 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503495+0000", 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.825e-05 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503611+0000", 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011631799999999999 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503644+0000", 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3476000000000003e-05 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.508954+0000", 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053093799999999998 2024-09-16T13:19:09.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.509019+0000", 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5425999999999993e-05 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1195 3.4 e101/23 3:24bef2c8:::rbd_data.112da28a57db.0000000000000020:27 v 101'326, mlcod=99'322)", 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 31.567393886000001, 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005296243, 2024-09-16T13:19:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373646+0000", 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7e-07 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373653+0000", 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4000000000000003e-06 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.008 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374730+0000", 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00107654 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374741+0000", 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1474999999999999e-05 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374829+0000", 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7584999999999996e-05 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.009 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374854+0000", 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5269000000000002e-05 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378893+0000", 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040386770000000001 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378942+0000", 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.010 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1197 3.4 e101/23 3:26d31b55:::rbd_data.112da28a57db.000000000000000c:27 v 101'328, mlcod=99'322)", 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 31.566276813000002, 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046376150000000003, 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374763+0000", 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5600000000000005e-07 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374784+0000", 2024-09-16T13:19:09.012 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0781999999999999e-05 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374794+0000", 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0558e-05 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374800+0000", 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375389+0000", 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058970500000000005 2024-09-16T13:19:09.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375407+0000", 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7801000000000001e-05 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379351+0000", 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039435030000000001 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379400+0000", 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9332000000000001e-05 2024-09-16T13:19:09.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:89341 3.4 e101/23 3:21ab973d:::rbd_data.112da28a57db.0000000000000002:head v 101'329, mlcod=101'326)", 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 31.443817020000001, 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005014859, 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:09.015 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497223+0000", 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8599999999999998e-07 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.016 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497231+0000", 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6709999999999999e-06 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497245+0000", 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.489e-05 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497253+0000", 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6199999999999999e-06 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.017 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498106+0000", 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00085257599999999996 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498130+0000", 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4776000000000001e-05 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502154+0000", 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004023703 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.018 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502237+0000", 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2936999999999996e-05 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.4.0:2718 3.9 e106/23 3:9702159d:::rbd_data.112d404789fd.000000000000001c:22 v 106'440, mlcod=105'437)", 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 26.519998875999999, 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063154409999999998, 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421041+0000", 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421055+0000", 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4579999999999999e-05 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421080+0000", 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4859e-05 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421089+0000", 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8950000000000004e-06 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421166+0000", 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6892999999999997e-05 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421197+0000", 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1325000000000002e-05 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427304+0000", 2024-09-16T13:19:09.022 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061062449999999997 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427356+0000", 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2197000000000002e-05 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.4.0:2792 3.9 e110/23 3:938f1173:::rbd_data.112d94eb720f.0000000000000006:20 v 110'460, mlcod=106'457)", 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 22.485428708000001, 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0041377590000000004, 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455611+0000", 2024-09-16T13:19:09.024 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2300000000000001e-07 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455621+0000", 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0064000000000001e-05 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455658+0000", 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6288000000000002e-05 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.025 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455678+0000", 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0812999999999999e-05 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455817+0000", 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013879800000000001 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.455886+0000", 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8842000000000003e-05 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:09.026 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.459651+0000", 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0037649519999999998 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:46.459748+0000", 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7379000000000004e-05 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1263 3.4 e119/23 3:2111795e:::rbd_data.112d3e228c72.0000000000000003:18 v 119'349, mlcod=118'345)", 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:55.540473+0000", 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 13.400565882, 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039574179999999999, 2024-09-16T13:19:09.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540473+0000", 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540473+0000", 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.028 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540474+0000", 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3200000000000001e-07 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540484+0000", 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0479000000000001e-05 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540494+0000", 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3910000000000004e-06 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.029 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540501+0000", 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7950000000000008e-06 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540593+0000", 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.1841000000000004e-05 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.540622+0000", 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.8597000000000001e-05 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.544386+0000", 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003763944 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:55.544431+0000", 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5039e-05 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.031 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:09.033 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:19:09.033 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:19:09.033 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:19:09.033 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:19:09.033 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:19:09.034 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:09.025+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:19:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:19:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:19:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:19:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:19:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:19:09.035 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:19:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:19:09.057 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:19:09.134 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:09.125+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:09.136 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:09.137 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:19:09.137 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:09.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:19:09.234 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:09.229+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:09.244 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:09.245 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:09.245 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:19:09.245 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:19:09.245 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:09.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:19:09.335 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:09.327+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:09.401 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62278 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:19:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081350430999997, 2024-09-16T13:19:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027907498999999999, 2024-09-16T13:19:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.402 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62278 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315827+0000", 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0600000000000001e-07 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315829+0000", 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.79e-06 2024-09-16T13:19:09.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315835+0000", 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.682e-06 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315838+0000", 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0730000000000001e-06 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.405 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343640+0000", 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027802384999999999 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343683+0000", 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3058e-05 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343734+0000", 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1104999999999999e-05 2024-09-16T13:19:09.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081335869999997, 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.409 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.410 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:19:09.411 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081311904000003, 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:19:09.413 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:19:09.414 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081296733999999, 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.416 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:19:09.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:19:09.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081282311000002, 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:19:09.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.421 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.422 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081261066000003, 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.425 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.426 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.427 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:19:09.428 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081246497000002, 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:19:09.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:19:09.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.432 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081227728000002, 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.434 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:19:09.435 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.436 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.437 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081212833000002, 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:19:09.439 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:19:09.440 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:19:09.441 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081198622000002, 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:19:09.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.443 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:19:09.445 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081185253000001, 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:19:09.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.448 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:19:09.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:19:09.450 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081170571000001, 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:19:09.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.452 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:19:09.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:19:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081155402, 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:19:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081140761999997, 2024-09-16T13:19:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081110033999998, 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:19:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081089016999996, 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:19:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:19:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:19:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:19:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:19:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081068909000003, 2024-09-16T13:19:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:19:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:19:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.477 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081049006000001, 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.480 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:19:09.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:19:09.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081033793000003, 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:19:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.485 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:19:09.486 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:19:09.487 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.081018258999997, 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:09.490 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:09.492 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:09.493 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:09.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:19:09.495 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:09.487+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:09.596 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:09.587+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:09.605 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:09.605 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:09.605 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:09.606 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:09.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:19:09.697 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:09.691+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:09.718 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:09.718 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:09.718 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:09.718 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:09.718 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:09.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:19:09.798 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:09.791+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:09.876 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.662493556000001, 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75320 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1114112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.41725383, 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048391086999999999, 2024-09-16T13:19:09.882 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75320 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455024+0000", 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455026+0000", 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:19:09.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455033+0000", 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8630000000000002e-06 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455037+0000", 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.608e-06 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503178+0000", 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048141646000000003 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503284+0000", 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105917 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503415+0000", 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130359 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.886 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75321 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1441792~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.417236838000001, 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048644984000000002, 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75321 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.887 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455041+0000", 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-16T13:19:09.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455043+0000", 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0480000000000001e-06 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455048+0000", 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455052+0000", 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000002e-06 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503461+0000", 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048409079000000001 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503580+0000", 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119234 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503685+0000", 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105006 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75322 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 2064384~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.417211440999999, 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048835044000000001, 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:09.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75322 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2599999999999998e-07 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455068+0000", 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.092e-06 2024-09-16T13:19:09.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455074+0000", 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.925e-06 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455078+0000", 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503740+0000", 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048661514000000003 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503822+0000", 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2196999999999999e-05 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503901+0000", 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9091000000000006e-05 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.895 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75344 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 589824~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416707801000001, 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048412121000000002, 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75344 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.896 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455570+0000", 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455572+0000", 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2749999999999998e-06 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455578+0000", 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455582+0000", 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4789999999999998e-06 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503719+0000", 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048137054999999998 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503819+0000", 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9938000000000004e-05 2024-09-16T13:19:09.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503982+0000", 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016225500000000001 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75345 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 917504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:19:09.900 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416691135000001, 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048702503000000001, 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75345 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:19:09.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455587+0000", 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455589+0000", 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455594+0000", 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6840000000000001e-06 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455598+0000", 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7009999999999998e-06 2024-09-16T13:19:09.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504027+0000", 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048429382999999999 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504123+0000", 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5251e-05 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504289+0000", 2024-09-16T13:19:09.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016599200000000001 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75346 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1212416~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416675777999998, 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048896186000000001, 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.905 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75346 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:19:09.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455602+0000", 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2799999999999996e-07 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455604+0000", 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455609+0000", 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455615+0000", 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2020000000000003e-06 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504315+0000", 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048700565000000001 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504382+0000", 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7088999999999994e-05 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504498+0000", 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115408 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.909 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75347 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1540096~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416657231999999, 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049121102999999999, 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75347 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7100000000000002e-07 2024-09-16T13:19:09.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455623+0000", 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455629+0000", 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455633+0000", 2024-09-16T13:19:09.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9400000000000004e-06 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504546+0000", 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048912718000000001 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504623+0000", 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7726999999999996e-05 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504741+0000", 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117874 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75353 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 98304~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416521488000001, 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048929180000000003, 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.914 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75353 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455758+0000", 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8789999999999999e-06 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455763+0000", 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2190000000000003e-06 2024-09-16T13:19:09.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455767+0000", 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504529+0000", 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048761566999999999 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504587+0000", 2024-09-16T13:19:09.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8564999999999999e-05 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504685+0000", 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7754e-05 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75354 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 393216~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416486417999998, 2024-09-16T13:19:09.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049086520000000002, 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75354 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455794+0000", 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.508e-06 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455801+0000", 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5630000000000003e-06 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455805+0000", 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:19:09.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504704+0000", 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048898981000000001 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504759+0000", 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4330000000000003e-05 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504877+0000", 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118608 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75355 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416467785999998, 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118086999999998, 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.923 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75355 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455810+0000", 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455812+0000", 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455818+0000", 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455821+0000", 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504914+0000", 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049092961999999997 2024-09-16T13:19:09.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504928+0000", 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3210000000000001e-05 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75356 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416452024999998, 2024-09-16T13:19:09.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049129843999999999, 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75356 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:19:09.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455826+0000", 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0399999999999996e-07 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455828+0000", 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-06 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455835+0000", 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8410000000000002e-06 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455839+0000", 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2400000000000001e-06 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504947+0000", 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049108546000000003 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504955+0000", 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6109999999999996e-06 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75357 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416434760999998, 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049141246, 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75357 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455843+0000", 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999998e-07 2024-09-16T13:19:09.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455845+0000", 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999998e-06 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455852+0000", 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455856+0000", 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6059999999999999e-06 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504974+0000", 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118385000000001 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504984+0000", 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7880000000000001e-06 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75358 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416417811999999, 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049149627000000001, 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.936 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75358 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455860+0000", 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455862+0000", 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:19:09.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455867+0000", 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.395e-06 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455873+0000", 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0120000000000002e-06 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505002+0000", 2024-09-16T13:19:09.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049128988999999998 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505009+0000", 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7950000000000002e-06 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75370 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:19:09.940 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416400058999997, 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049174830000000003, 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75370 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:19:09.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455878+0000", 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455880+0000", 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455885+0000", 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455889+0000", 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6090000000000001e-06 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505040+0000", 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049151242999999997 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505052+0000", 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2055999999999999e-05 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75371 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416384923000003, 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049181864999999998, 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75371 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455893+0000", 2024-09-16T13:19:09.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455895+0000", 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455900+0000", 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.536e-06 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455904+0000", 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505067+0000", 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049163224999999998 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505074+0000", 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.948 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75372 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416369785000001, 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049214870000000001, 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75372 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.949 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455908+0000", 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.89e-07 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455910+0000", 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455915+0000", 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:19:09.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455919+0000", 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.472e-06 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505109+0000", 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049190747 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505122+0000", 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3009e-05 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75373 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416355267999997, 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049224933999999998, 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:09.953 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75373 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:09.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455924+0000", 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455930+0000", 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4929999999999998e-06 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455936+0000", 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4459999999999998e-06 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505140+0000", 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049203510999999998 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505147+0000", 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1670000000000002e-06 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75374 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416337188999996, 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049235427999999998, 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.957 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75374 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455946+0000", 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860000000000001e-06 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455953+0000", 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5119999999999997e-06 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455956+0000", 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.579e-06 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505166+0000", 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049209527000000003 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505175+0000", 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75375 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.416316909000003, 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049243375999999998, 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:09.961 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75375 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:09.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455961+0000", 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455963+0000", 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729999999999999e-06 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455970+0000", 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2520000000000002e-06 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455975+0000", 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505198+0000", 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049222852999999997 2024-09-16T13:19:09.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:09.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:09.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:09.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505204+0000", 2024-09-16T13:19:09.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:19:09.965 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.965 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.965 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.965 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:09.965 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:09.965 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:09.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:19:09.967 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:09.961+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:10.067 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:10.059+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:10.075 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:10.075 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:10.075 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:10.075 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:10.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:19:10.169 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:10.163+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:10.180 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:10.180 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:10.180 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:10.181 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:10.181 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:10.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:19:10.269 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:10.261+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 69.465014647999993, 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.292 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:19:10.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:19:10.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82800 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1130496~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428956552999999, 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050353660000000001, 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.299 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.300 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.300 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82800 2024-09-16T13:19:10.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.300 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:19:10.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:19:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858530+0000", 2024-09-16T13:19:10.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858534+0000", 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8229999999999997e-06 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858547+0000", 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3302000000000001e-05 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858557+0000", 2024-09-16T13:19:10.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-05 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908637+0000", 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050079811000000002 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908731+0000", 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4085000000000005e-05 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908883+0000", 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000151676 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82801 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1212416~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428920658000003, 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050667642999999998, 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82801 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858566+0000", 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3499999999999995e-07 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858570+0000", 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8299999999999998e-06 2024-09-16T13:19:10.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858590+0000", 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9831e-05 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858597+0000", 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6729999999999992e-06 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908930+0000", 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050332492999999999 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909036+0000", 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010603999999999999 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909233+0000", 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019704100000000001 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.308 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82802 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428880362000001, 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050929965000000001, 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82802 2024-09-16T13:19:10.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:10.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858610+0000", 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0820000000000001e-06 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858655+0000", 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4901000000000003e-05 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858668+0000", 2024-09-16T13:19:10.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2343e-05 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909278+0000", 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050610005 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909374+0000", 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6399999999999999e-05 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909535+0000", 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016137599999999999 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82803 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1474560~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428771511000001, 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051051886999999997, 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.313 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82803 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:19:10.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858715+0000", 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999995e-07 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858719+0000", 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.732e-06 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858728+0000", 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6730000000000004e-06 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858733+0000", 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.532e-06 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909572+0000", 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050839399 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909647+0000", 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5073000000000004e-05 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909766+0000", 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118805 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82804 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1654784~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428747139000002, 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051320085000000001, 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82804 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3399999999999998e-07 2024-09-16T13:19:10.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858742+0000", 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4150000000000002e-06 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858748+0000", 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0090000000000004e-06 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858753+0000", 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1649999999999997e-06 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909812+0000", 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051058612000000003 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909907+0000", 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5795e-05 2024-09-16T13:19:10.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910059+0000", 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015135499999999999 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82805 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1736704~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428727223000003, 2024-09-16T13:19:10.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051834002999999997, 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82805 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858762+0000", 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4959999999999999e-06 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858768+0000", 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8830000000000003e-06 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858771+0000", 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9319999999999998e-06 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910078+0000", 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051306696999999998 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910163+0000", 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5065e-05 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910593+0000", 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042940799999999999 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82806 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1916928~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428710442000003, 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052120149999999997, 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.327 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82806 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858776+0000", 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2600000000000002e-07 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858778+0000", 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3070000000000001e-06 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858784+0000", 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858788+0000", 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8870000000000002e-06 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910643+0000", 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051854355999999997 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910735+0000", 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2454000000000006e-05 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910896+0000", 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016055600000000001 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82807 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1998848~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428691274000002, 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052345503000000002, 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82807 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.332 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:10.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858797+0000", 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858804+0000", 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0079999999999996e-06 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858808+0000", 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910935+0000", 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052126695000000001 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911010+0000", 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4764000000000004e-05 2024-09-16T13:19:10.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911140+0000", 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013022500000000001 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82830 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 770048~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428673263999997, 2024-09-16T13:19:10.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052616234999999997, 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82830 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:10.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858815+0000", 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3130000000000001e-06 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858821+0000", 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7200000000000003e-06 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858838+0000", 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6492000000000001e-05 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911178+0000", 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052340684999999998 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911272+0000", 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4106999999999999e-05 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911429+0000", 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156389 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.340 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82831 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428639476999997, 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052803142999999997, 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82831 2024-09-16T13:19:10.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858848+0000", 2024-09-16T13:19:10.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.209e-06 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858858+0000", 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0111e-05 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858870+0000", 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1976e-05 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858876+0000", 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5649999999999998e-06 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911449+0000", 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052573110999999999 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911521+0000", 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1209000000000002e-05 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911650+0000", 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012896199999999999 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82832 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1032192~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428601815999997, 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053031663999999999, 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.345 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82832 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858885+0000", 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6899999999999996e-07 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858889+0000", 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.087e-06 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858905+0000", 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5744999999999999e-05 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858909+0000", 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7269999999999998e-06 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911673+0000", 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052763697999999998 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911767+0000", 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4054000000000004e-05 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911916+0000", 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148584 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82833 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1114112~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.349 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428568712999997, 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053225354000000002, 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82833 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:19:10.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858918+0000", 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858922+0000", 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.371e-06 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858933+0000", 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1076e-05 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858940+0000", 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9840000000000004e-06 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911944+0000", 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053004044 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912021+0000", 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7189999999999998e-05 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912143+0000", 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012160200000000001 2024-09-16T13:19:10.353 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82834 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1294336~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428538961000001, 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053469475000000002, 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.354 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82834 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858948+0000", 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858952+0000", 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858970+0000", 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.889e-05 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858978+0000", 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912172+0000", 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053193670999999998 2024-09-16T13:19:10.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912268+0000", 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6100000000000005e-05 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912416+0000", 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014817500000000001 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.358 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82835 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1376256~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428498667, 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053671286999999998, 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82835 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.359 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858988+0000", 2024-09-16T13:19:10.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858992+0000", 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858999+0000", 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859003+0000", 2024-09-16T13:19:10.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912456+0000", 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053453374999999997 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912526+0000", 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9237000000000001e-05 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912658+0000", 2024-09-16T13:19:10.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013284000000000001 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82836 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1556480~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428478820000002, 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053937880000000001, 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.363 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82836 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859008+0000", 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859010+0000", 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1950000000000002e-06 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859016+0000", 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8429999999999998e-06 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859019+0000", 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912697+0000", 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053677322 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912796+0000", 2024-09-16T13:19:10.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8991000000000001e-05 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912945+0000", 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149207 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82837 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1638400~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428462797000002, 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054154068999999999, 2024-09-16T13:19:10.367 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82837 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859024+0000", 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859026+0000", 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:19:10.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859031+0000", 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6690000000000003e-06 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859035+0000", 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6239999999999999e-06 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912981+0000", 2024-09-16T13:19:10.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053946409000000001 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913054+0000", 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3052999999999993e-05 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913177+0000", 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122803 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.372 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82838 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1818624~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.372 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:10.372 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428447359000003, 2024-09-16T13:19:10.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054416554999999998, 2024-09-16T13:19:10.372 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.372 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82838 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859041+0000", 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2139999999999999e-06 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859047+0000", 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6359999999999999e-06 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859051+0000", 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6519999999999999e-06 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913215+0000", 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054164344000000003 2024-09-16T13:19:10.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913307+0000", 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2546999999999993e-05 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913455+0000", 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147639 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82839 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1900544~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428431697000001, 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054634963000000002, 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82839 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.377 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859055+0000", 2024-09-16T13:19:10.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859057+0000", 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3910000000000001e-06 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859066+0000", 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5150000000000006e-06 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859072+0000", 2024-09-16T13:19:10.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5810000000000002e-06 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913493+0000", 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054420751000000003 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913563+0000", 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9938999999999996e-05 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913689+0000", 2024-09-16T13:19:10.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012626299999999999 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82840 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 2080768~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.428404661000002, 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054884993999999999, 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.381 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82840 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859082+0000", 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5799999999999998e-07 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859086+0000", 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7510000000000002e-06 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859097+0000", 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1085e-05 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859104+0000", 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-06 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913718+0000", 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054614085 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913814+0000", 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5514999999999996e-05 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913966+0000", 2024-09-16T13:19:10.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015253 2024-09-16T13:19:10.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.385 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.385 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.385 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:10.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:19:10.387 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:10.385+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:10.487 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:10.479+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:10.491 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:10.491 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:10.491 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:10.491 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:10.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:19:10.589 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:10.583+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:10.600 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:10.600 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:10.600 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:10.601 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:10.601 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:10.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:19:10.689 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:10.685+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:10.715 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:10.715 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.829112889000001, 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.716 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:19:10.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.720 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.829094053999995, 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:10.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:19:10.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.829074915000007, 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:19:10.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:19:10.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.829057856999995, 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:19:10.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.829040496999994, 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:19:10.733 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.737 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.829022788000003, 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:10.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:19:10.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.829005488999996, 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:19:10.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:19:10.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:19:10.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828970390999999, 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.746 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:19:10.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:19:10.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828947236000005, 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:19:10.750 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:10.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.754 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828906216999997, 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:19:10.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828883317000006, 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.759 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:19:10.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828858268000005, 2024-09-16T13:19:10.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.767 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828833047000003, 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:19:10.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:10.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828814682000001, 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.772 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:19:10.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:19:10.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:19:10.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.776 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828551544999996, 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.777 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:19:10.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:10.781 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828526537000002, 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:19:10.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.785 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828498205000002, 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.786 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:19:10.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.789 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828209506999997, 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.790 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:19:10.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:19:10.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:19:10.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.828189397000003, 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.794 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:19:10.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.823988473, 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:10.798 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:10.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:19:10.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:19:10.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:10.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:10.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:10.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:19:10.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:19:10.802 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.802 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.802 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.802 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:10.802 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:10.802 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:10.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:19:10.804 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:10.801+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:10.904 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:10.899+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:10.911 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:10.911 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:10.911 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:10.911 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:10.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:19:11.005 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:11.001+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:11.022 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:11.022 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:11.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:19:11.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:19:11.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:19:11.105 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:11.099+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90616 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:head [delete] snapc 21=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.572230412, 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.870938027, 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.148 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90616 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570781+0000", 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.88e-06 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570792+0000", 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0567000000000001e-05 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570816+0000", 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-05 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570850+0000", 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3779999999999998e-05 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570946+0000", 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6294999999999999e-05 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:11.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570958+0000", 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438058+0000", 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86709994199999996 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438302+0000", 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024435100000000002 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:19:11.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438940+0000", 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063796000000000002 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441545+0000", 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026055509999999998 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441576+0000", 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0247e-05 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441622+0000", 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6047999999999997e-05 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441716+0000", 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4524000000000005e-05 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90621 3.e 3.2a047d2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.572129172, 2024-09-16T13:19:11.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.860052445, 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90621 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570881+0000", 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570885+0000", 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7230000000000004e-06 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570901+0000", 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5756999999999998e-05 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570908+0000", 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2960000000000001e-06 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571040+0000", 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131803 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571069+0000", 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8694e-05 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430917+0000", 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85984788899999998 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430932+0000", 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5129e-05 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.158 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90624 3.e 3.2071842e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.571545908000001, 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85950820299999997, 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90624 2024-09-16T13:19:11.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571465+0000", 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1799999999999999e-06 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571490+0000", 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5320000000000002e-05 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571524+0000", 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4190000000000003e-05 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571543+0000", 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8674000000000001e-05 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571647+0000", 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103803 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571669+0000", 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2354e-05 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430962+0000", 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85929225799999998 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430971+0000", 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4239999999999999e-06 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2775 3.1b e108/107 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:23 v 108'605, mlcod=108'601)", 2024-09-16T13:19:11.163 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 26.580835672999999, 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069960880000000001, 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:11.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562174+0000", 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7000000000000004e-07 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562181+0000", 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2459999999999997e-06 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562194+0000", 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2945000000000001e-05 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562200+0000", 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3679999999999998e-06 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562237+0000", 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7039999999999998e-05 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562262+0000", 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4443999999999998e-05 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569096+0000", 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068345039999999999 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569169+0000", 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2770999999999998e-05 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(3.12 109/108 [PushOp(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22, version: 83'227, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22@83'227, size: 1048576, copy_subset: [0~1048576], clone_subset: {}, snapset: 22=[]:{22=[22]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.482700008999998, 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056147790000000003, 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660310+0000", 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-06 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662684+0000", 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023738169999999999 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662716+0000", 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2271999999999999e-05 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662753+0000", 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6851999999999999e-05 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662882+0000", 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012939700000000001 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662892+0000", 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0315e-05 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.665923+0000", 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030311140000000001 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90825 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch reconnect cookie 140083147222288 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.691994957999999, 2024-09-16T13:19:11.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0248692960000001, 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90825 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451018+0000", 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8569999999999997e-06 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451032+0000", 2024-09-16T13:19:11.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4586999999999999e-05 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451078+0000", 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6180000000000002e-05 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451101+0000", 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2773e-05 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451261+0000", 2024-09-16T13:19:11.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016003600000000001 2024-09-16T13:19:11.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:11.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451276+0000", 2024-09-16T13:19:11.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4387e-05 2024-09-16T13:19:11.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459307+0000", 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080310969999999992 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459318+0000", 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1136e-05 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471312+0000", 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.011993798 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471508+0000", 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019619700000000001 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.475883+0000", 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004375248 2024-09-16T13:19:11.177 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2793 3.1b e110/107 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 v 110'613, mlcod=108'609)", 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.686832137, 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070642279999999997, 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.178 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456178+0000", 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.517e-06 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456230+0000", 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2323e-05 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456251+0000", 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0619999999999999e-05 2024-09-16T13:19:11.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456269+0000", 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7883000000000001e-05 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456632+0000", 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036355300000000002 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456686+0000", 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3454e-05 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463210+0000", 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065238550000000003 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463241+0000", 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1022999999999998e-05 2024-09-16T13:19:11.182 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90826 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.686767880000001, 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023049584, 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90826 2024-09-16T13:19:11.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456244+0000", 2024-09-16T13:19:11.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3000000000000002e-06 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456261+0000", 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7036000000000001e-05 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456294+0000", 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456313+0000", 2024-09-16T13:19:11.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8876999999999999e-05 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456444+0000", 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013131300000000001 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456462+0000", 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7920000000000001e-05 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459343+0000", 2024-09-16T13:19:11.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028811539999999999 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459351+0000", 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5870000000000004e-06 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472631+0000", 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0132794970000001 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472737+0000", 2024-09-16T13:19:11.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106496 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.473418+0000", 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068136 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.474875+0000", 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001456564 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479175+0000", 2024-09-16T13:19:11.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042996450000000004 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479238+0000", 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3610999999999998e-05 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479290+0000", 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2163999999999997e-05 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.189 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1617 3.12 e111/110 3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22 v 111'330, mlcod=0'0)", 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.677038058000001, 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060955660000000002, 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:11.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465972+0000", 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2999999999999999e-07 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465984+0000", 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2133999999999999e-05 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465998+0000", 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4184e-05 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466009+0000", 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1371e-05 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467605+0000", 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015953129999999999 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467632+0000", 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7447e-05 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472017+0000", 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043852589999999999 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472066+0000", 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8928000000000003e-05 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1618 3.12 e111/110 3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22 v 111'332, mlcod=0'0)", 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.676441402000002, 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00616418, 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466568+0000", 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6000000000000003e-07 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466575+0000", 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8809999999999998e-06 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466899+0000", 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032458399999999999 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466912+0000", 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2767e-05 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469208+0000", 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022958369999999998 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469231+0000", 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2578e-05 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472690+0000", 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003459443 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472731+0000", 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1430000000000001e-05 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1621 3.12 e111/110 3:4a3469d0:::rbd_data.112d12abb20f.000000000000003a:28 v 111'338, mlcod=111'334)", 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.654533840999999, 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005747411, 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488476+0000", 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488497+0000", 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1542000000000001e-05 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488508+0000", 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0461e-05 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488514+0000", 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2990000000000002e-06 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488568+0000", 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4559999999999999e-05 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488591+0000", 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2416e-05 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494170+0000", 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055793279999999997 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494222+0000", 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2079000000000001e-05 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.201 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1622 3.12 e111/110 3:498ca772:::rbd_data.112d12abb20f.0000000000000017:28 v 111'340, mlcod=111'334)", 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.653514185999999, 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074768710000000004, 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489503+0000", 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8720000000000002e-06 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489515+0000", 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2191000000000001e-05 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489524+0000", 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6170000000000007e-06 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489592+0000", 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7670999999999996e-05 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489614+0000", 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2112000000000001e-05 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496931+0000", 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073173739999999998 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496971+0000", 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0247999999999998e-05 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90832 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch unwatch cookie 140083147222288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.648221443000001, 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055530110000000001, 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90832 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494787+0000", 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494788+0000", 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3900000000000004e-07 2024-09-16T13:19:11.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494791+0000", 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9050000000000001e-06 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494799+0000", 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3100000000000001e-06 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494804+0000", 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2170000000000002e-06 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496030+0000", 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012255689999999999 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496148+0000", 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117612 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496408+0000", 2024-09-16T13:19:11.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026060999999999999 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.499840+0000", 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034319670000000002 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500173+0000", 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033327399999999999 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500232+0000", 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8303000000000001e-05 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.500340+0000", 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010860499999999999 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.211 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:91826 3.11 3:8c261a44:::rbd_data.112d770f4ae2.000000000000000e:head [delete] snapc 1c=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.496437299, 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077560520000000003, 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 91826 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646576+0000", 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2699999999999998e-06 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646592+0000", 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6439999999999998e-05 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646637+0000", 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5120000000000002e-05 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646657+0000", 2024-09-16T13:19:11.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9737000000000001e-05 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646782+0000", 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012466699999999999 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646790+0000", 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6680000000000007e-06 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651149+0000", 2024-09-16T13:19:11.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043590959999999998 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651538+0000", 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000389071 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651714+0000", 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017587200000000001 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:11.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654242+0000", 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025272659999999998 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654249+0000", 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.61e-06 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654294+0000", 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4644000000000001e-05 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654327+0000", 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3590999999999997e-05 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92599 3.11 3:8b1d31ff:::rbd_data.112d3e228c72.0000000000000004:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.503636867999999, 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070407359999999997, 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.218 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92599 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639375+0000", 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9399999999999998e-06 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639386+0000", 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1606999999999999e-05 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639426+0000", 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9329000000000002e-05 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639439+0000", 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.348e-05 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639642+0000", 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000202976 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639865+0000", 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022277500000000001 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640355+0000", 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049025000000000002 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.644293+0000", 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003937739 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646196+0000", 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019030410000000001 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646332+0000", 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013587899999999999 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646413+0000", 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.072e-05 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92601 3.13 3:cbdfcc30:::rbd_data.112d3e228c72.0000000000000006:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.503535228000001, 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069350899999999997, 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92601 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639475+0000", 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.59e-06 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639482+0000", 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.46e-06 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639568+0000", 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6064999999999997e-05 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639591+0000", 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3564e-05 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639945+0000", 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035337399999999999 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640456+0000", 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051193699999999996 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640736+0000", 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027973199999999998 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.643326+0000", 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025895129999999999 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646146+0000", 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028207359999999999 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646277+0000", 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013046999999999999 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646409+0000", 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013164900000000001 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92605 3.11 3:8caec98f:::rbd_data.112d3e228c72.000000000000000a:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.501100457, 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067373449999999996, 2024-09-16T13:19:11.229 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92605 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641911+0000", 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-06 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641925+0000", 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.403e-05 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641959+0000", 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3448999999999998e-05 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641976+0000", 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7397e-05 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.642102+0000", 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012618899999999999 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.642511+0000", 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040815799999999999 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.643089+0000", 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057817299999999997 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.644631+0000", 2024-09-16T13:19:11.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015421969999999999 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648481+0000", 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038498930000000001 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648543+0000", 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2477000000000005e-05 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648646+0000", 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010228199999999999 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:93417 3.e 3:74f3d54d:::rbd_data.112d30ff5030.000000000000000b:head [delete] snapc 13=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 11.045896416, 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077949680000000002, 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.235 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 93417 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:11.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097116+0000", 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3859999999999999e-06 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097141+0000", 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5732e-05 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097191+0000", 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9854000000000001e-05 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097211+0000", 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0046000000000001e-05 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097304+0000", 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2318999999999995e-05 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:11.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097311+0000", 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4089999999999997e-06 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.101746+0000", 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044344780000000004 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.102087+0000", 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034153299999999998 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:11.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.102361+0000", 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027436799999999999 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104058+0000", 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001696431 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104805+0000", 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0007473 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104848+0000", 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3112999999999999e-05 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104907+0000", 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8999000000000002e-05 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:94264 3.13 3:ccbe4cb7:::rbd_id.image_client.0-clone10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:05.111021+0000", 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 6.0319876250000002, 2024-09-16T13:19:11.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068979260000000004, 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 94264 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111021+0000", 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111021+0000", 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111024+0000", 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111036+0000", 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2046e-05 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111071+0000", 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4898999999999999e-05 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111090+0000", 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9196999999999999e-05 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111231+0000", 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014067600000000001 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111243+0000", 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2174000000000001e-05 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.115107+0000", 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038635219999999999 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.115225+0000", 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011879099999999999 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.115395+0000", 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016947899999999999 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117212+0000", 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001817319 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117788+0000", 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057622499999999998 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117862+0000", 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3218000000000003e-05 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117919+0000", 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7429000000000002e-05 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:94516 3.13 3:ceb7d054:::rbd_header.112d21cb8818:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:09.057925+0000", 2024-09-16T13:19:11.247 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 2.0850834699999998, 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075387060000000001, 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 94516 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057925+0000", 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057925+0000", 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057928+0000", 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.526e-06 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057939+0000", 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1283e-05 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057961+0000", 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1962000000000001e-05 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057985+0000", 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3867999999999999e-05 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.058984+0000", 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00099875000000000007 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.059268+0000", 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028392199999999998 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.060097+0000", 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00082984400000000005 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.064948+0000", 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048506900000000004 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.065340+0000", 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039161300000000002 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.065389+0000", 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9063000000000001e-05 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.065464+0000", 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5185000000000006e-05 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.252 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:11.253 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:11.253 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:11.254 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:19:11.254 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:19:11.254 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:19:11.254 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:11.247+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:11.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:19:11.281 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:19:11.318 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:19:11.342 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:19:11.354 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:11.349+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:11.364 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:19:11.376 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:11.376 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:11.385 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:19:11.387 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:19:11.387 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:19:11.388 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:19:11.388 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:19:11.388 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:19:11.388 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:19:11.388 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:19:11.388 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:19:11.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:19:11.456 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:11.449+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:11.534 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:11.534 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:11.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:19:11.557 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:11.551+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:11.639 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:11.640 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:11.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:19:11.659 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:11.651+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:11.746 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:11.746 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:11.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:19:11.759 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:11.755+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:11.849 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:11.849 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:11.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:19:11.859 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:11.853+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:11.879 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:19:11.879 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:19:11.879 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T13:19:11.879 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-09-16T13:19:11.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0 2024-09-16T13:19:11.961 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:11.953+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:11.966 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:11.967 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:11.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:19:12.061 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:12.053+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:19:12.144 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:12.144 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:12.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:19:12.162 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:12.157+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:12.264 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:12.257+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:12.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:19:12.365 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:12.363+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:12.413 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:12.413 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:12.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:19:12.465 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:12.457+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:12.521 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:12.522 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:12.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:19:12.566 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:12.561+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:12.629 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:12.629 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:12.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:19:12.667 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:12.659+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:12.768 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:12.763+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:12.784 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:12.784 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:12.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:19:12.869 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:12.865+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:12.906 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:12.906 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:12.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:19:12.974 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:12.965+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.6 primary-affinity to 0 (802) 2024-09-16T13:19:13.017 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:13.017 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:13.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:19:13.071 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:13.065+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:13.124 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:13.124 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:13.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:19:13.171 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:13.165+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:13.227 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:13.227 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:13.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:19:13.273 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:13.267+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:13.336 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:13.336 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:13.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:19:13.373 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:13.365+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:13.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:19:13.474 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:13.465+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:13.498 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:13.499 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:13.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:19:13.548 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:19:13.569 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:19:13.575 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:13.567+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:13.597 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:19:13.606 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:13.607 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:13.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:19:13.617 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:19:13.639 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:19:13.659 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:19:13.676 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:13.668+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:13.682 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:19:13.703 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:19:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:19:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:19:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:19:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:19:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:19:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:19:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:19:13.706 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:19:13.767 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:13.767 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:13.767 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:13.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:19:13.777 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:13.768+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:13.877 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:13.868+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:13.879 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:13.879 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.992148, 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:19:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:19:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:19:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:19:13.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.992129164999994, 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:19:13.885 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:13.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:13.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.890 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.992110026000006, 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:19:13.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.895 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.992092967999994, 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.896 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:19:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:19:13.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:19:13.900 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.992075607999993, 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.901 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:19:13.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:19:13.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.992057899000002, 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.906 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:19:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:19:13.910 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.992040599999996, 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.911 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:19:13.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.915 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.992005501999998, 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:13.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:19:13.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991982347000004, 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:19:13.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:19:13.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.925 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991941327999996, 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:19:13.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991918428000005, 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.931 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:19:13.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:19:13.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:19:13.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:19:13.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:19:13.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.935 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991893379000004, 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.936 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:13.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:19:13.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.940 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991868158000003, 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.941 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:19:13.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991849793, 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:19:13.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:13.950 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991586655999996, 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:19:13.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991561648000001, 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.955 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:19:13.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.959 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991533316000002, 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:13.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:19:13.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991244617999996, 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.965 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:19:13.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:19:13.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.991224508000002, 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.970 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:19:13.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:19:13.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:19:13.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:19:13.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:19:13.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:19:13.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:13.975 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.987023583999999, 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:13.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:19:13.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:19:13.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:13.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:13.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:13.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:19:13.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:19:13.980 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.980 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.980 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.980 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:13.980 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:13.980 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:13.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:19:13.982 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:13.972+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:14.077 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:14.077 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:14.077 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:14.077 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:14.083 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:14.075+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:14.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:19:14.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T13:19:14.184 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:14.176+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:14.189 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:14.189 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:14.189 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:14.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:19:14.285 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:14.279+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:14.300 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:14.300 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:14.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:14.300 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:14.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.300 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90616 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:head [delete] snapc 21=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:14.300 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.724295174000002, 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.870938027, 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90616 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570781+0000", 2024-09-16T13:19:14.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.88e-06 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570792+0000", 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0567000000000001e-05 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570816+0000", 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-05 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570850+0000", 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3779999999999998e-05 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570946+0000", 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6294999999999999e-05 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570958+0000", 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:19:14.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438058+0000", 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86709994199999996 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438302+0000", 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024435100000000002 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438940+0000", 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063796000000000002 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441545+0000", 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026055509999999998 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441576+0000", 2024-09-16T13:19:14.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0247e-05 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441622+0000", 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6047999999999997e-05 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441716+0000", 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4524000000000005e-05 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.307 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90621 3.e 3.2a047d2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.724193933999999, 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.860052445, 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.308 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90621 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570881+0000", 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570885+0000", 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7230000000000004e-06 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570901+0000", 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5756999999999998e-05 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570908+0000", 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2960000000000001e-06 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571040+0000", 2024-09-16T13:19:14.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131803 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571069+0000", 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8694e-05 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430917+0000", 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85984788899999998 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430932+0000", 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5129e-05 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90624 3.e 3.2071842e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.723610669999999, 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85950820299999997, 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.313 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90624 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571465+0000", 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1799999999999999e-06 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571490+0000", 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5320000000000002e-05 2024-09-16T13:19:14.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571524+0000", 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4190000000000003e-05 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571543+0000", 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8674000000000001e-05 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571647+0000", 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103803 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571669+0000", 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2354e-05 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430962+0000", 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85929225799999998 2024-09-16T13:19:14.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430971+0000", 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4239999999999999e-06 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2775 3.1b e108/107 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:23 v 108'605, mlcod=108'601)", 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:14.318 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.732900435000001, 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069960880000000001, 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:14.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562174+0000", 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7000000000000004e-07 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562181+0000", 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2459999999999997e-06 2024-09-16T13:19:14.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562194+0000", 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2945000000000001e-05 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562200+0000", 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3679999999999998e-06 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562237+0000", 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7039999999999998e-05 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562262+0000", 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4443999999999998e-05 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569096+0000", 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068345039999999999 2024-09-16T13:19:14.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569169+0000", 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2770999999999998e-05 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(3.12 109/108 [PushOp(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22, version: 83'227, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22@83'227, size: 1048576, copy_subset: [0~1048576], clone_subset: {}, snapset: 22=[]:{22=[22]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-16T13:19:14.323 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.634764771, 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056147790000000003, 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660309+0000", 2024-09-16T13:19:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.660310+0000", 2024-09-16T13:19:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-06 2024-09-16T13:19:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662684+0000", 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023738169999999999 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662716+0000", 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2271999999999999e-05 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662753+0000", 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6851999999999999e-05 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662882+0000", 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012939700000000001 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.662892+0000", 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0315e-05 2024-09-16T13:19:14.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:45.665923+0000", 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030311140000000001 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90825 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch reconnect cookie 140083147222288 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:19:14.328 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.844059720000001, 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0248692960000001, 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90825 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451018+0000", 2024-09-16T13:19:14.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8569999999999997e-06 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451032+0000", 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4586999999999999e-05 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451078+0000", 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6180000000000002e-05 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451101+0000", 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2773e-05 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451261+0000", 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016003600000000001 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:14.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451276+0000", 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4387e-05 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459307+0000", 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080310969999999992 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459318+0000", 2024-09-16T13:19:14.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1136e-05 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471312+0000", 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.011993798 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471508+0000", 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019619700000000001 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.475883+0000", 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004375248 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2793 3.1b e110/107 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 v 110'613, mlcod=108'609)", 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.838896899000002, 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070642279999999997, 2024-09-16T13:19:14.335 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:14.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456178+0000", 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.517e-06 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456230+0000", 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2323e-05 2024-09-16T13:19:14.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456251+0000", 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0619999999999999e-05 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456269+0000", 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7883000000000001e-05 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456632+0000", 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036355300000000002 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456686+0000", 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3454e-05 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463210+0000", 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065238550000000003 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463241+0000", 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1022999999999998e-05 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90826 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 27.838832642, 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023049584, 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90826 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.341 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456244+0000", 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3000000000000002e-06 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456261+0000", 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7036000000000001e-05 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456294+0000", 2024-09-16T13:19:14.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456313+0000", 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8876999999999999e-05 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456444+0000", 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013131300000000001 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456462+0000", 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7920000000000001e-05 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459343+0000", 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028811539999999999 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459351+0000", 2024-09-16T13:19:14.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5870000000000004e-06 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472631+0000", 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0132794970000001 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472737+0000", 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106496 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.473418+0000", 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068136 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.474875+0000", 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001456564 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:14.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479175+0000", 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042996450000000004 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479238+0000", 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3610999999999998e-05 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479290+0000", 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2163999999999997e-05 2024-09-16T13:19:14.348 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1617 3.12 e111/110 3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22 v 111'330, mlcod=0'0)", 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 26.829102819999999, 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060955660000000002, 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.349 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465972+0000", 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2999999999999999e-07 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465984+0000", 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2133999999999999e-05 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465998+0000", 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4184e-05 2024-09-16T13:19:14.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466009+0000", 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1371e-05 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467605+0000", 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015953129999999999 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467632+0000", 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7447e-05 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472017+0000", 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043852589999999999 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472066+0000", 2024-09-16T13:19:14.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8928000000000003e-05 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1618 3.12 e111/110 3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22 v 111'332, mlcod=0'0)", 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 26.828506164, 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00616418, 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.354 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466568+0000", 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6000000000000003e-07 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466575+0000", 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8809999999999998e-06 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466899+0000", 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032458399999999999 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466912+0000", 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2767e-05 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469208+0000", 2024-09-16T13:19:14.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022958369999999998 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469231+0000", 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2578e-05 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472690+0000", 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003459443 2024-09-16T13:19:14.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472731+0000", 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1430000000000001e-05 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1621 3.12 e111/110 3:4a3469d0:::rbd_data.112d12abb20f.000000000000003a:28 v 111'338, mlcod=111'334)", 2024-09-16T13:19:14.359 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 26.806598603000001, 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005747411, 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488475+0000", 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488476+0000", 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488497+0000", 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1542000000000001e-05 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488508+0000", 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0461e-05 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488514+0000", 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2990000000000002e-06 2024-09-16T13:19:14.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488568+0000", 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4559999999999999e-05 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.488591+0000", 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2416e-05 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494170+0000", 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055793279999999997 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.494222+0000", 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2079000000000001e-05 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1622 3.12 e111/110 3:498ca772:::rbd_data.112d12abb20f.0000000000000017:28 v 111'340, mlcod=111'334)", 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 26.805578948000001, 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074768710000000004, 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489503+0000", 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8720000000000002e-06 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489515+0000", 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2191000000000001e-05 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489524+0000", 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6170000000000007e-06 2024-09-16T13:19:14.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489592+0000", 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7670999999999996e-05 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489614+0000", 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2112000000000001e-05 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496931+0000", 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073173739999999998 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496971+0000", 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0247999999999998e-05 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:91826 3.11 3:8c261a44:::rbd_data.112d770f4ae2.000000000000000e:head [delete] snapc 1c=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.648502060999999, 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077560520000000003, 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 91826 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.370 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646576+0000", 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2699999999999998e-06 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646592+0000", 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6439999999999998e-05 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646637+0000", 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5120000000000002e-05 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646657+0000", 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9737000000000001e-05 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646782+0000", 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012466699999999999 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646790+0000", 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6680000000000007e-06 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651149+0000", 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043590959999999998 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651538+0000", 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000389071 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651714+0000", 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017587200000000001 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654242+0000", 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025272659999999998 2024-09-16T13:19:14.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654249+0000", 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.61e-06 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654294+0000", 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4644000000000001e-05 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654327+0000", 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3590999999999997e-05 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92599 3.11 3:8b1d31ff:::rbd_data.112d3e228c72.0000000000000004:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.655701629999999, 2024-09-16T13:19:14.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070407359999999997, 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92599 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:14.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639375+0000", 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9399999999999998e-06 2024-09-16T13:19:14.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639386+0000", 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1606999999999999e-05 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639426+0000", 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9329000000000002e-05 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639439+0000", 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.348e-05 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639642+0000", 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000202976 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639865+0000", 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022277500000000001 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640355+0000", 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049025000000000002 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.644293+0000", 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003937739 2024-09-16T13:19:14.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646196+0000", 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019030410000000001 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646332+0000", 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013587899999999999 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646413+0000", 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.072e-05 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92601 3.13 3:cbdfcc30:::rbd_data.112d3e228c72.0000000000000006:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.655599989999999, 2024-09-16T13:19:14.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069350899999999997, 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92601 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:14.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639475+0000", 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.59e-06 2024-09-16T13:19:14.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639482+0000", 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.46e-06 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639568+0000", 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6064999999999997e-05 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639591+0000", 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3564e-05 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639945+0000", 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035337399999999999 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640456+0000", 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051193699999999996 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640736+0000", 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027973199999999998 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.643326+0000", 2024-09-16T13:19:14.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025895129999999999 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646146+0000", 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028207359999999999 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646277+0000", 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013046999999999999 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646409+0000", 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013164900000000001 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92605 3.11 3:8caec98f:::rbd_data.112d3e228c72.000000000000000a:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.653165219000002, 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067373449999999996, 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92605 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:14.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641911+0000", 2024-09-16T13:19:14.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-06 2024-09-16T13:19:14.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641925+0000", 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.403e-05 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641959+0000", 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3448999999999998e-05 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641976+0000", 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7397e-05 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.642102+0000", 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012618899999999999 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.642511+0000", 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040815799999999999 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:14.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.643089+0000", 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057817299999999997 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.644631+0000", 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015421969999999999 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648481+0000", 2024-09-16T13:19:14.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038498930000000001 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648543+0000", 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2477000000000005e-05 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648646+0000", 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010228199999999999 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.397 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:93417 3.e 3:74f3d54d:::rbd_data.112d30ff5030.000000000000000b:head [delete] snapc 13=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 14.197961178, 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077949680000000002, 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.398 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 93417 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:14.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097116+0000", 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3859999999999999e-06 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097141+0000", 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5732e-05 2024-09-16T13:19:14.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097191+0000", 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9854000000000001e-05 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097211+0000", 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0046000000000001e-05 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097304+0000", 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2318999999999995e-05 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097311+0000", 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4089999999999997e-06 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.101746+0000", 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044344780000000004 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.102087+0000", 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034153299999999998 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.102361+0000", 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027436799999999999 2024-09-16T13:19:14.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:14.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104058+0000", 2024-09-16T13:19:14.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001696431 2024-09-16T13:19:14.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:14.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104805+0000", 2024-09-16T13:19:14.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0007473 2024-09-16T13:19:14.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104848+0000", 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3112999999999999e-05 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104907+0000", 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8999000000000002e-05 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.405 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:94264 3.13 3:ccbe4cb7:::rbd_id.image_client.0-clone10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:05.111021+0000", 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.1840523869999995, 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068979260000000004, 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 94264 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111021+0000", 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111021+0000", 2024-09-16T13:19:14.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111024+0000", 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111036+0000", 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2046e-05 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111071+0000", 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4898999999999999e-05 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111090+0000", 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9196999999999999e-05 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111231+0000", 2024-09-16T13:19:14.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014067600000000001 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111243+0000", 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2174000000000001e-05 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.115107+0000", 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038635219999999999 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.115225+0000", 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011879099999999999 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.115395+0000", 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016947899999999999 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117212+0000", 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001817319 2024-09-16T13:19:14.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117788+0000", 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057622499999999998 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117862+0000", 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3218000000000003e-05 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117919+0000", 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7429000000000002e-05 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:94516 3.13 3:ceb7d054:::rbd_header.112d21cb8818:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:09.057925+0000", 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.237148232, 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075387060000000001, 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 94516 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057925+0000", 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057925+0000", 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057928+0000", 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.526e-06 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057939+0000", 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1283e-05 2024-09-16T13:19:14.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057961+0000", 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1962000000000001e-05 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057985+0000", 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3867999999999999e-05 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.058984+0000", 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00099875000000000007 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.059268+0000", 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028392199999999998 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.060097+0000", 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00082984400000000005 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.064948+0000", 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048506900000000004 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.065340+0000", 2024-09-16T13:19:14.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039161300000000002 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.065389+0000", 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9063000000000001e-05 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.065464+0000", 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5185000000000006e-05 2024-09-16T13:19:14.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.420 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.420 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.5.0:2212 3.16 e135/107 3:6d453e39:::rbd_data.112ddef65b5a.000000000000006b:b v 135'402, mlcod=120'397)", 2024-09-16T13:19:14.420 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:11.068462+0000", 2024-09-16T13:19:14.420 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.2266114579999998, 2024-09-16T13:19:14.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005881346, 2024-09-16T13:19:14.420 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:14.420 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068462+0000", 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068462+0000", 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:14.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068463+0000", 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.178e-06 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068474+0000", 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0382e-05 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068492+0000", 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8831999999999999e-05 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068502+0000", 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0066e-05 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:14.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.069980+0000", 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001477892 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.070006+0000", 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5978000000000001e-05 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.074244+0000", 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042378729999999996 2024-09-16T13:19:14.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:14.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:14.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:14.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.074343+0000", 2024-09-16T13:19:14.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9145e-05 2024-09-16T13:19:14.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.425 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:14.425 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:14.425 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:14.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:19:14.428 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:14.420+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:14.528 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:14.523+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:14.530 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:14.530 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:14.530 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:19:14.530 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:14.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:19:14.630 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:14.620+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:14.640 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:14.640 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:14.640 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:19:14.641 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:19:14.641 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:14.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:19:14.730 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:14.724+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:14.795 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66873 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4034560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:19:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174794863000002, 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034635435999999999, 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66873 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:19:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615784+0000", 2024-09-16T13:19:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615785+0000", 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615786+0000", 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7829999999999999e-06 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615792+0000", 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1900000000000003e-06 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615795+0000", 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9450000000000002e-06 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650232+0000", 2024-09-16T13:19:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034437669999999997 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650301+0000", 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8329000000000003e-05 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650420+0000", 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000119132 2024-09-16T13:19:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66874 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4042752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174761656999998, 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034827871000000003, 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.801 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66874 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615818+0000", 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615820+0000", 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.063e-06 2024-09-16T13:19:14.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615826+0000", 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9739999999999999e-06 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615835+0000", 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5920000000000001e-06 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.804 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650445+0000", 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034610100999999997 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650512+0000", 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650645+0000", 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013316399999999999 2024-09-16T13:19:14.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66875 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4050944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174740747999998, 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032338000000003, 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.806 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66875 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615838+0000", 2024-09-16T13:19:14.807 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615839+0000", 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615841+0000", 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9250000000000002e-06 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615846+0000", 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.711e-06 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615851+0000", 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9579999999999998e-06 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650676+0000", 2024-09-16T13:19:14.809 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.034824524000000003 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650758+0000", 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1547999999999999e-05 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650871+0000", 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000113244 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66876 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4059136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174723853000003, 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035239591000000001, 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66876 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615855+0000", 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615856+0000", 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8700000000000001e-07 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615858+0000", 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.857e-06 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.813 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615863+0000", 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5389999999999998e-06 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615866+0000", 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650899+0000", 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035032448000000001 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.650973+0000", 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3627000000000005e-05 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651095+0000", 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012239400000000001 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66877 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4067328~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174709153000002, 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035445691000000001, 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66877 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.816 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.817 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615870+0000", 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9200000000000002e-07 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615872+0000", 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.703e-06 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615877+0000", 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2650000000000004e-06 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615881+0000", 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.292e-06 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651123+0000", 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035242373 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.819 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651198+0000", 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5096000000000001e-05 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651316+0000", 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011757 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.820 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66878 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4075520~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.17469526, 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035647261999999999, 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66878 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.821 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.822 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615884+0000", 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5400000000000002e-07 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615890+0000", 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6189999999999999e-06 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615896+0000", 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6529999999999999e-06 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.823 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615900+0000", 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0069999999999999e-06 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651337+0000", 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035437877999999999 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.824 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651395+0000", 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7219999999999998e-05 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651531+0000", 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013653100000000001 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.825 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66879 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.826 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:19:14.826 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174675919000002, 2024-09-16T13:19:14.826 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035838241999999999, 2024-09-16T13:19:14.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.826 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.826 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.826 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.826 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.826 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66879 2024-09-16T13:19:14.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615903+0000", 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615904+0000", 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9599999999999999e-07 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615906+0000", 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7889999999999999e-06 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.828 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615911+0000", 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6679999999999998e-06 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615914+0000", 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.151e-06 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651558+0000", 2024-09-16T13:19:14.829 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035643672000000001 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651624+0000", 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6070999999999998e-05 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651742+0000", 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011729499999999999 2024-09-16T13:19:14.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66880 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4091904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174661338999996, 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036042148000000003, 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.831 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66880 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:19:14.832 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615918+0000", 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2099999999999997e-07 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615924+0000", 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.1059999999999996e-06 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615930+0000", 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6250000000000004e-06 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615933+0000", 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3950000000000001e-06 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.834 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651773+0000", 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035839579000000003 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651846+0000", 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2602999999999996e-05 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651960+0000", 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000114319 2024-09-16T13:19:14.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66881 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4100096~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174640322000002, 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036242563999999998, 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.836 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66881 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615939+0000", 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0200000000000003e-07 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.838 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615941+0000", 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.72e-06 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615946+0000", 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.259e-06 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615950+0000", 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:19:14.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.651985+0000", 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036035821000000003 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652058+0000", 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2651999999999995e-05 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652181+0000", 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123428 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66882 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4108288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174626291999999, 2024-09-16T13:19:14.841 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036437628999999999, 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66882 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:19:14.842 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615953+0000", 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7300000000000002e-07 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615955+0000", 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8199999999999999e-06 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615961+0000", 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2349999999999996e-06 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.844 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615965+0000", 2024-09-16T13:19:14.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4620000000000001e-06 2024-09-16T13:19:14.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652207+0000", 2024-09-16T13:19:14.845 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036241873000000001 2024-09-16T13:19:14.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.845 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.845 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652276+0000", 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9105999999999999e-05 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652391+0000", 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011476000000000001 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66883 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4116480~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.846 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174611069000001, 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036642620000000001, 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66883 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.847 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615968+0000", 2024-09-16T13:19:14.848 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.77e-07 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615970+0000", 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7519999999999999e-06 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615977+0000", 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9349999999999997e-06 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.849 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.615985+0000", 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1389999999999995e-06 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652416+0000", 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036430381999999997 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652487+0000", 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1698000000000004e-05 2024-09-16T13:19:14.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652611+0000", 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000123337 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66884 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4124672~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.851 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174346642000003, 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036613363000000003, 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66884 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.852 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.853 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616233+0000", 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.16e-07 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616237+0000", 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5719999999999999e-06 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616245+0000", 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.8820000000000001e-06 2024-09-16T13:19:14.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616251+0000", 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9630000000000003e-06 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652638+0000", 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036387705999999999 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.855 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652719+0000", 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0469999999999994e-05 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652846+0000", 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000126954 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66885 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4132864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174324089999999, 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036785929000000002, 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66885 2024-09-16T13:19:14.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616255+0000", 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0999999999999999e-07 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616258+0000", 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0269999999999998e-06 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616263+0000", 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8760000000000002e-06 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616267+0000", 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4979999999999998e-06 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652873+0000", 2024-09-16T13:19:14.860 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036606376000000003 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.652956+0000", 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2824000000000005e-05 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653041+0000", 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.4918000000000002e-05 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66886 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4141056~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174307661, 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078279999999998, 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.862 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66886 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:19:14.863 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616272+0000", 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616274+0000", 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.73e-06 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616279+0000", 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7169999999999996e-06 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616283+0000", 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1420000000000002e-06 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.865 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653119+0000", 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036836016999999999 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653204+0000", 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5575999999999995e-05 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.866 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653350+0000", 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014569499999999999 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66887 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4149248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174293157999998, 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037308557999999999, 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.867 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66887 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616286+0000", 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.84e-07 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.869 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616288+0000", 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7099999999999999e-06 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616294+0000", 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4029999999999998e-06 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616297+0000", 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0079999999999998e-06 2024-09-16T13:19:14.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653375+0000", 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037078883 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653455+0000", 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0034e-05 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.871 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653595+0000", 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013913599999999999 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66888 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4157440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174278786999999, 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537323999999997, 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.872 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66888 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616300+0000", 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616301+0000", 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616302+0000", 2024-09-16T13:19:14.874 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.584e-06 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616308+0000", 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3870000000000003e-06 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616317+0000", 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2229999999999996e-06 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.875 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653620+0000", 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037302572999999999 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653701+0000", 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1771e-05 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.876 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653838+0000", 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013640599999999999 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66889 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4165632~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174258258999998, 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037755019000000001, 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.877 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66889 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616321+0000", 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2300000000000002e-07 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.879 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616323+0000", 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.905e-06 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616329+0000", 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4639999999999997e-06 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.880 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616332+0000", 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0180000000000002e-06 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653862+0000", 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037529973000000001 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.653937+0000", 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4954000000000001e-05 2024-09-16T13:19:14.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654076+0000", 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013928200000000001 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66890 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4173824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.882 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174242853000003, 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037917073000000003, 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66890 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616336+0000", 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.884 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616337+0000", 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616338+0000", 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7689999999999999e-06 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.885 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616371+0000", 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2806999999999999e-05 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616375+0000", 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.827e-06 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654115+0000", 2024-09-16T13:19:14.886 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.03773986 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654181+0000", 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5652000000000001e-05 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654253+0000", 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2789999999999999e-05 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.887 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66891 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4182016~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174194982000003, 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038160044999999997, 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.888 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.889 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.889 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66891 2024-09-16T13:19:14.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.889 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:19:14.890 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.890 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.890 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616384+0000", 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616385+0000", 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8400000000000004e-07 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616387+0000", 2024-09-16T13:19:14.891 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.356e-06 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616393+0000", 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9519999999999999e-06 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616396+0000", 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2959999999999999e-06 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654302+0000", 2024-09-16T13:19:14.892 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037905031999999998 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654395+0000", 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.3560000000000005e-05 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654544+0000", 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014916500000000001 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:66892 3.12 3:4a0da337:::rbd_data.112d420e934f.0000000000000004:21 [read 4190208~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.174179170999999, 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038342319999999999, 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 66892 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.894 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.895 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616400+0000", 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616407+0000", 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4060000000000002e-06 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616412+0000", 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6400000000000002e-06 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.896 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.616419+0000", 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.037e-06 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654571+0000", 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038152835000000003 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654625+0000", 2024-09-16T13:19:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3643000000000001e-05 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.654742+0000", 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000117348 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:14.898 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:14.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:19:14.900 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:14.891+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:15.001 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:14.995+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:15.005 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:15.005 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:15.005 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:19:15.006 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:15.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:19:15.105 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:15.096+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:15.105 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is set 2024-09-16T13:19:15.116 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:19:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:19:15.116 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:15.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:19:15.206 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:15.200+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:15.222 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:15.222 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:15.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:15.222 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:15.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.222 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64860 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3899392~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.222 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:19:15.222 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655870938, 2024-09-16T13:19:15.222 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035719773000000003, 2024-09-16T13:19:15.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.223 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.223 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.223 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64860 2024-09-16T13:19:15.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.223 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.223 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563334+0000", 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9399999999999995e-07 2024-09-16T13:19:15.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563336+0000", 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563342+0000", 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3240000000000002e-06 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563345+0000", 2024-09-16T13:19:15.225 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4199999999999999e-06 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.598973+0000", 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035627631999999999 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599006+0000", 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3495999999999999e-05 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599054+0000", 2024-09-16T13:19:15.226 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7460999999999998e-05 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64861 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3907584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655856057000001, 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035815461, 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.227 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64861 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563349+0000", 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4900000000000001e-07 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563351+0000", 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.854e-06 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.229 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563356+0000", 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5940000000000001e-06 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563360+0000", 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4060000000000001e-06 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599076+0000", 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035716171999999997 2024-09-16T13:19:15.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599116+0000", 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9777999999999998e-05 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599164+0000", 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8307999999999999e-05 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64862 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655841463999998, 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035954097999999997, 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64862 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.232 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563363+0000", 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563364+0000", 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8000000000000001e-07 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.233 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563365+0000", 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.821e-06 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563371+0000", 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0769999999999998e-06 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563374+0000", 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3560000000000001e-06 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.234 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599187+0000", 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035813153 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599252+0000", 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4873e-05 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599317+0000", 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5438000000000006e-05 2024-09-16T13:19:15.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64863 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3923968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655815910000001, 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036051842000000001, 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.236 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64863 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563389+0000", 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0699999999999998e-07 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563392+0000", 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3140000000000002e-06 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563398+0000", 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.066e-06 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563402+0000", 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9859999999999996e-06 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599339+0000", 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.035937678000000001 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599377+0000", 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:19:15.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599441+0000", 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3380999999999995e-05 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64864 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3932160~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655799506000001, 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036187015000000003, 2024-09-16T13:19:15.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64864 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.241 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563405+0000", 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563406+0000", 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5199999999999998e-07 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563411+0000", 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4480000000000002e-06 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563417+0000", 2024-09-16T13:19:15.242 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8089999999999998e-06 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563421+0000", 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0110000000000002e-06 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599458+0000", 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036036870999999998 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599515+0000", 2024-09-16T13:19:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7067000000000001e-05 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599592+0000", 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7457000000000001e-05 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64865 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3940352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655780116000003, 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036265720000000001, 2024-09-16T13:19:15.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64865 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563425+0000", 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3799999999999998e-07 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563427+0000", 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.001e-06 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563433+0000", 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5489999999999997e-06 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563436+0000", 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.484e-06 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599613+0000", 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036176815000000001 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599646+0000", 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3192999999999999e-05 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599690+0000", 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4240000000000003e-05 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64866 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3948544~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.248 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655764337999997, 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036355749999999999, 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64866 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:19:15.249 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563440+0000", 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563441+0000", 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563443+0000", 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0870000000000001e-06 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563448+0000", 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.062e-06 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563451+0000", 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4470000000000002e-06 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599712+0000", 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036261007999999997 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599753+0000", 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0476999999999999e-05 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599796+0000", 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3226000000000003e-05 2024-09-16T13:19:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64867 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3956736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655749598, 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036484356000000003, 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64867 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563455+0000", 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563456+0000", 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5299999999999999e-07 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563458+0000", 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9980000000000002e-06 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563463+0000", 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3909999999999997e-06 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563466+0000", 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3759999999999999e-06 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599810+0000", 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036343683000000002 2024-09-16T13:19:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599864+0000", 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3565e-05 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599939+0000", 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.5889999999999993e-05 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64868 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3964928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655734440000003, 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036573319999999999, 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64868 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563470+0000", 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563471+0000", 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.1100000000000001e-07 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563473+0000", 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563479+0000", 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.178e-06 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563483+0000", 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6339999999999999e-06 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.599963+0000", 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036480090999999999 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600001+0000", 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8340999999999998e-05 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600044+0000", 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2472999999999998e-05 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64869 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3973120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655718466000003, 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036707818000000003, 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64869 2024-09-16T13:19:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563486+0000", 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563487+0000", 2024-09-16T13:19:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9100000000000003e-07 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563489+0000", 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8989999999999999e-06 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563494+0000", 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5820000000000001e-06 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563498+0000", 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.174e-06 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600061+0000", 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036563127000000001 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600119+0000", 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8476000000000001e-05 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600194+0000", 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4869000000000004e-05 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64870 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3981312~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655703424000002, 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036852528000000002, 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64870 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563501+0000", 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563502+0000", 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6600000000000002e-07 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563504+0000", 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9479999999999998e-06 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563509+0000", 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4360000000000001e-06 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.268 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563512+0000", 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3239999999999999e-06 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600218+0000", 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036705812999999997 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600280+0000", 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2298000000000006e-05 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.269 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600354+0000", 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3343000000000006e-05 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64871 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3989504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655688718, 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036938311000000001, 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.270 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64871 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563516+0000", 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5699999999999998e-07 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563518+0000", 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9130000000000001e-06 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563523+0000", 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2120000000000002e-06 2024-09-16T13:19:15.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563529+0000", 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0710000000000002e-06 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600376+0000", 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036847958 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600411+0000", 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4388999999999999e-05 2024-09-16T13:19:15.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600454+0000", 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.3411000000000002e-05 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64872 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 3997696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655672211000002, 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037077038999999999, 2024-09-16T13:19:15.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64872 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.275 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563532+0000", 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563533+0000", 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6599999999999996e-07 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563535+0000", 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1270000000000001e-06 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.276 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563540+0000", 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3120000000000001e-06 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563544+0000", 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4699999999999998e-06 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600476+0000", 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.036931691000000003 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.277 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600543+0000", 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7475e-05 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600610+0000", 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6397999999999997e-05 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64873 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4005888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:19:15.278 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655657185999999, 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037239282999999998, 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64873 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563548+0000", 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563550+0000", 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9939999999999999e-06 2024-09-16T13:19:15.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563555+0000", 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2449999999999998e-06 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563558+0000", 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2009999999999999e-06 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600630+0000", 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037071933000000001 2024-09-16T13:19:15.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600702+0000", 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1245e-05 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600787+0000", 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5290000000000005e-05 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64874 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4014080~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655642739999998, 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037394846000000002, 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64874 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.283 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563562+0000", 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0400000000000002e-07 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.284 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563565+0000", 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1900000000000002e-06 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563570+0000", 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4199999999999998e-06 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563574+0000", 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0779999999999997e-06 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.285 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600812+0000", 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037237600000000003 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600876+0000", 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4690000000000006e-05 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600957+0000", 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.0463999999999994e-05 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64875 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4022272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655626945000002, 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037537883000000001, 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64875 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563578+0000", 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.65e-07 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563580+0000", 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1739999999999999e-06 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563587+0000", 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2589999999999997e-06 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563591+0000", 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8969999999999999e-06 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.289 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.600982+0000", 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037390117 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601044+0000", 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2788999999999998e-05 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601116+0000", 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1282e-05 2024-09-16T13:19:15.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64876 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4030464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.65560962, 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037682571999999998, 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.291 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64876 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563595+0000", 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563596+0000", 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4799999999999999e-07 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563613+0000", 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7935999999999999e-05 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563620+0000", 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4740000000000002e-06 2024-09-16T13:19:15.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563624+0000", 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6210000000000001e-06 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601141+0000", 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037517170000000002 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.294 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601203+0000", 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2187999999999995e-05 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601278+0000", 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4734999999999995e-05 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64877 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4038656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:19:15.295 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655576011000001, 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037813404000000002, 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64877 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563629+0000", 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563631+0000", 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1330000000000002e-06 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563637+0000", 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2959999999999998e-06 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563640+0000", 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3189999999999999e-06 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601304+0000", 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037664438000000001 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601372+0000", 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7618000000000002e-05 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601442+0000", 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0149999999999998e-05 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64878 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4046848~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:19:15.299 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655546837999999, 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037956456, 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64878 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563658+0000", 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563659+0000", 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4899999999999995e-07 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563661+0000", 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.249e-06 2024-09-16T13:19:15.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563667+0000", 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3470000000000004e-06 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563671+0000", 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7170000000000002e-06 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601473+0000", 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037801955999999998 2024-09-16T13:19:15.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601537+0000", 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3960000000000004e-05 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601614+0000", 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:64879 3.4 3:20f84b8f:::rbd_data.112d420e934f.0000000000000001:21 [read 4055040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:15.303 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 61.655530198999998, 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.038485017000000003, 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 64879 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563675+0000", 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4099999999999998e-07 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563677+0000", 2024-09-16T13:19:15.305 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1299999999999999e-06 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563682+0000", 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.0649999999999998e-06 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.563688+0000", 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.3329999999999996e-06 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601640+0000", 2024-09-16T13:19:15.306 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.037951864000000002 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.601705+0000", 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5573000000000004e-05 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.602160+0000", 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000454411 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.308 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.308 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:15.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:19:15.309 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:15.300+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:15.410 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:15.404+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:15.457 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:15.457 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:15.457 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:19:15.458 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:15.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:19:15.511 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:15.507+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:15.564 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:15.564 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:15.564 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:19:15.564 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:19:15.564 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:15.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:19:15.618 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:15.612+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:15.676 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:15.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 131.66386951699999, 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 131.66270259500001, 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.684 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 123.114364759, 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:15.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:19:15.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:19:15.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 123.114225119, 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:19:15.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 123.113724589, 2024-09-16T13:19:15.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:19:15.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.697 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 118.118555203, 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.698 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.699 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.700 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 107.862141362, 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.702 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.703 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.704 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 107.249578255, 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:15.706 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:19:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:19:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.411472085, 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:19:15.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.711 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:19:15.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:19:15.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:19:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 80.047344752000001, 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:19:15.716 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.717 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:75420 3.4 e88/23 3:2605e3b8:::rbd_object_map.112d9326a5dd.0000000000000026:head v 88'165, mlcod=88'163)", 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 56.168615736, 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055875079999999997, 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:15.719 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503433+0000", 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503451+0000", 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7640999999999999e-05 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.720 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503476+0000", 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5599999999999999e-05 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503495+0000", 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.825e-05 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503611+0000", 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011631799999999999 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.721 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503644+0000", 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3476000000000003e-05 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.508954+0000", 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053093799999999998 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.509019+0000", 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5425999999999993e-05 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1195 3.4 e101/23 3:24bef2c8:::rbd_data.112da28a57db.0000000000000020:27 v 101'326, mlcod=99'322)", 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.298402109999998, 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005296243, 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.723 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373646+0000", 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7e-07 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373653+0000", 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4000000000000003e-06 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.724 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374730+0000", 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00107654 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374741+0000", 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1474999999999999e-05 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374829+0000", 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7584999999999996e-05 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.725 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374854+0000", 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5269000000000002e-05 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378893+0000", 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040386770000000001 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378942+0000", 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.726 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1197 3.4 e101/23 3:26d31b55:::rbd_data.112da28a57db.000000000000000c:27 v 101'328, mlcod=99'322)", 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.297285037000002, 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046376150000000003, 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374763+0000", 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5600000000000005e-07 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374784+0000", 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0781999999999999e-05 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.728 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374794+0000", 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0558e-05 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374800+0000", 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375389+0000", 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058970500000000005 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.729 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375407+0000", 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7801000000000001e-05 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379351+0000", 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039435030000000001 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379400+0000", 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9332000000000001e-05 2024-09-16T13:19:15.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:89341 3.4 e101/23 3:21ab973d:::rbd_data.112da28a57db.0000000000000002:head v 101'329, mlcod=101'326)", 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 38.174825243999997, 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005014859, 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497223+0000", 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8599999999999998e-07 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497231+0000", 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6709999999999999e-06 2024-09-16T13:19:15.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497245+0000", 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.489e-05 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497253+0000", 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6199999999999999e-06 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498106+0000", 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00085257599999999996 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498130+0000", 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4776000000000001e-05 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502154+0000", 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004023703 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502237+0000", 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2936999999999996e-05 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.734 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.4.0:2718 3.9 e106/23 3:9702159d:::rbd_data.112d404789fd.000000000000001c:22 v 106'440, mlcod=105'437)", 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 33.251007100000002, 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063154409999999998, 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421041+0000", 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421055+0000", 2024-09-16T13:19:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4579999999999999e-05 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421080+0000", 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4859e-05 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421089+0000", 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8950000000000004e-06 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.737 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421166+0000", 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6892999999999997e-05 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421197+0000", 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1325000000000002e-05 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427304+0000", 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061062449999999997 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.738 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427356+0000", 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2197000000000002e-05 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:94609 3.9 e133/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 133'502, mlcod=132'500)", 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:09.204270+0000", 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 6.4677778400000001, 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004587041, 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.739 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204270+0000", 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204270+0000", 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204270+0000", 2024-09-16T13:19:15.740 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0399999999999995e-07 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204304+0000", 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3590999999999997e-05 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204343+0000", 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8634999999999999e-05 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204358+0000", 2024-09-16T13:19:15.741 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5566999999999999e-05 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204468+0000", 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109862 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204526+0000", 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7841999999999999e-05 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.208761+0000", 2024-09-16T13:19:15.742 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042354469999999998 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.208857+0000", 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5393e-05 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:95238 3.4 e139/23 3:23de2965:::rbd_data.112d64676e7f.0000000000000007:head v 139'365, mlcod=136'362)", 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:15.246402+0000", 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.42564526400000002, 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051633470000000004, 2024-09-16T13:19:15.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246402+0000", 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246402+0000", 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.744 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246404+0000", 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246423+0000", 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9330999999999998e-05 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246461+0000", 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7128000000000001e-05 2024-09-16T13:19:15.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246477+0000", 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5926e-05 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246584+0000", 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107215 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246640+0000", 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6529e-05 2024-09-16T13:19:15.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251525+0000", 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048844149999999996 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251566+0000", 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0893999999999998e-05 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.747 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:95239 3.4 e139/23 3:249f6365:::rbd_data.112d64676e7f.0000000000000008:head v 139'366, mlcod=136'362)", 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:15.246826+0000", 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.425221766, 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047688360000000003, 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246826+0000", 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246826+0000", 2024-09-16T13:19:15.748 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246827+0000", 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246839+0000", 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1953999999999999e-05 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246860+0000", 2024-09-16T13:19:15.749 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1474000000000001e-05 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246872+0000", 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2109e-05 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.247634+0000", 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00076168299999999998 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.247679+0000", 2024-09-16T13:19:15.750 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5404000000000001e-05 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251570+0000", 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038906140000000001 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251595+0000", 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4443999999999998e-05 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.751 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:95247 3.4 e139/23 3:248e1f52:::rbd_data.112d64676e7f.0000000000000036:head v 139'368, mlcod=136'362)", 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:15.250634+0000", 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.42141397400000002, 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005299967, 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250634+0000", 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.752 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250634+0000", 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250634+0000", 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3799999999999996e-07 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250641+0000", 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2130000000000002e-06 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250657+0000", 2024-09-16T13:19:15.753 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5096000000000001e-05 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250664+0000", 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6879999999999999e-06 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250810+0000", 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014586699999999999 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250848+0000", 2024-09-16T13:19:15.754 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8220999999999999e-05 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.255895+0000", 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005047134 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.255933+0000", 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.755 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:95250 3.0 e139/23 3:040d935c:::rbd_data.112d64676e7f.0000000000000051:head v 139'418, mlcod=136'415)", 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:15.251287+0000", 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.42076038399999999, 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046599950000000001, 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251287+0000", 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251287+0000", 2024-09-16T13:19:15.756 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251288+0000", 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.85e-07 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251294+0000", 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3099999999999997e-06 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251308+0000", 2024-09-16T13:19:15.757 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3312000000000001e-05 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251315+0000", 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4170000000000003e-06 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251362+0000", 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7203000000000002e-05 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251378+0000", 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5899000000000001e-05 2024-09-16T13:19:15.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.255918+0000", 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045400600000000003 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.255947+0000", 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9009e-05 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:15.759 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:15.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T13:19:15.761 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:15.755+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:15.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:19:15.861 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:15.852+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:15.864 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:15.864 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:15.864 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:19:15.864 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:15.874 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:19:15.898 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:19:15.922 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:19:15.941 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:19:15.961 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:19:15.962 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:15.959+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:15.974 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:15.974 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:15.984 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:19:15.985 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:19:16.006 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:19:16.029 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:19:16.063 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:16.056+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:16.081 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:16.082 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:16.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:19:16.164 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:16.155+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:16.205 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:16.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:19:16.265 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:16.259+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:16.266 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:19:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:19:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:19:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:19:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:19:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:19:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:19:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:19:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:19:16.316 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:16.317 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:16.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:19:16.368 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:16.363+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:16.467 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:16.459+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:16.476 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:16.476 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:16.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:19:16.568 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:16.564+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:16.586 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:16.586 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:16.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:19:16.669 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:16.662+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:16.692 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:16.692 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:16.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:19:16.771 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:16.762+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:16.801 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:16.801 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:16.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:19:16.871 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:16.862+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:16.908 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:16.908 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:16.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:19:16.972 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:16.968+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:17.061 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:17.062 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:17.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:19:17.072 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:17.066+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:17.121 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T13:19:17.169 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:17.169 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:17.172 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:17.166+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:17.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:19:17.274 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:17.266+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:17.325 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:17.325 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:17.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:19:17.381 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:17.374+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:17.430 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:17.430 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:17.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:19:17.475 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:17.468+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:17.540 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:17.540 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:17.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T13:19:17.577 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:17.568+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:17.655 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:17.656 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:17.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T13:19:17.678 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:17.674+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:17.779 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:17.774+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:17.807 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:17.807 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:17.817 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T13:19:17.879 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:17.874+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:17.913 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:17.913 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:17.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T13:19:17.980 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is set 2024-09-16T13:19:17.980 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:17.972+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:17.992 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-16T13:19:17.992 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T13:19:17.993 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-16T13:19:17.993 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T13:19:17.993 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T13:19:17.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-16T13:19:18.041 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:18.041 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:18.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T13:19:18.081 INFO:tasks.ceph.osd.5.smithi151.stderr:2024-09-16T13:19:18.074+0000 7fb0dc252700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 22279) UID: 0 2024-09-16T13:19:18.091 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:18.091 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "" 2024-09-16T13:19:18.091 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:18.159 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:18.159 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:18.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T13:19:18.181 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:18.172+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:18.257 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:18.258 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:18.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T13:19:18.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T13:19:18.283 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:18.274+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:18.362 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:18.362 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:18.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T13:19:18.384 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:18.376+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:18.442 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0: {} 2024-09-16T13:19:18.461 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1: {} 2024-09-16T13:19:18.468 INFO:teuthology.orchestra.run.smithi098.stdout:ERROR: (22) Invalid argument 2024-09-16T13:19:18.468 INFO:teuthology.orchestra.run.smithi098.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T13:19:18.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T13:19:18.485 INFO:tasks.ceph.osd.7.smithi151.stderr:2024-09-16T13:19:18.478+0000 7f234fd14700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33048) UID: 0 2024-09-16T13:19:18.487 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2: {} 2024-09-16T13:19:18.511 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3: {} 2024-09-16T13:19:18.546 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4: {} 2024-09-16T13:19:18.580 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5: {} 2024-09-16T13:19:18.586 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-16T13:19:18.580+0000 7f2bc2123700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22258) UID: 0 2024-09-16T13:19:18.600 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:18.600 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:18.600 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:18.600 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:18.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.600 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2439 3.0 e31/23 3:02316a81:::rbd_data.112dbfd5d823.000000000000003b:head v 31'25, mlcod=31'23)", 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 134.58756231000001, 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055504229999999996, 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.601 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008178+0000", 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008179+0000", 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.99e-07 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.602 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008184+0000", 2024-09-16T13:19:18.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2079999999999999e-06 2024-09-16T13:19:18.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.603 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.603 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008198+0000", 2024-09-16T13:19:18.603 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3869000000000001e-05 2024-09-16T13:19:18.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008204+0000", 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.908e-06 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008281+0000", 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7188999999999996e-05 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.008317+0000", 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5673000000000002e-05 2024-09-16T13:19:18.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013682+0000", 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053651879999999999 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013728+0000", 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6689e-05 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.605 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.606 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:2442 3.0 e31/23 3:02a2635a:::rbd_data.112dbfd5d823.000000000000006a:head v 31'26, mlcod=31'23)", 2024-09-16T13:19:18.606 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:18.606 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 134.586395388, 2024-09-16T13:19:18.606 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044808419999999996, 2024-09-16T13:19:18.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.606 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009345+0000", 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.607 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009346+0000", 2024-09-16T13:19:18.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.7499999999999999e-07 2024-09-16T13:19:18.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009353+0000", 2024-09-16T13:19:18.608 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0280000000000003e-06 2024-09-16T13:19:18.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.608 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.608 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009368+0000", 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5515999999999998e-05 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009383+0000", 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4567e-05 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009442+0000", 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8771999999999999e-05 2024-09-16T13:19:18.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.009477+0000", 2024-09-16T13:19:18.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5302000000000002e-05 2024-09-16T13:19:18.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.610 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.610 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013788+0000", 2024-09-16T13:19:18.610 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0043108299999999999 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:04.013826+0000", 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8052000000000001e-05 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6969 3.4 e37/23 3:25aa93f3:::rbd_data.112d6fb8d387.00000000000001e5:head v 37'32, mlcod=37'29)", 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 126.038057552, 2024-09-16T13:19:18.611 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061758890000000004, 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557683+0000", 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557684+0000", 2024-09-16T13:19:18.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:19:18.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.613 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.613 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557721+0000", 2024-09-16T13:19:18.613 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6529000000000002e-05 2024-09-16T13:19:18.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557745+0000", 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4675999999999999e-05 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557759+0000", 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3169999999999999e-05 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.614 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557937+0000", 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00017891900000000001 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558009+0000", 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1705000000000005e-05 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563757+0000", 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0057482319999999998 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.615 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563859+0000", 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000101241 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6966 3.0 e37/23 3:01b4a689:::rbd_data.112d6fb8d387.0000000000000189:head v 37'48, mlcod=37'45)", 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 126.037917912, 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0058744909999999999, 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.616 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557822+0000", 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.617 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557823+0000", 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.097e-06 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557833+0000", 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.4960000000000002e-06 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557853+0000", 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9579999999999999e-05 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.618 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557876+0000", 2024-09-16T13:19:18.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3122e-05 2024-09-16T13:19:18.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.557999+0000", 2024-09-16T13:19:18.619 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00012302799999999999 2024-09-16T13:19:18.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.619 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.619 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558054+0000", 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5098e-05 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563669+0000", 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005615391 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563697+0000", 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7679e-05 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:6970 3.9 e37/23 3:9120b033:::rbd_data.112d6fb8d387.00000000000001fc:head v 37'33, mlcod=37'31)", 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 126.037417382, 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056094159999999999, 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558323+0000", 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558324+0000", 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.545e-06 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.622 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558339+0000", 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4114000000000001e-05 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558364+0000", 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5694000000000002e-05 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558379+0000", 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4523e-05 2024-09-16T13:19:18.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558522+0000", 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014289899999999999 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.558569+0000", 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7271000000000002e-05 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563850+0000", 2024-09-16T13:19:18.624 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0052805960000000002 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:12.563932+0000", 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2774000000000004e-05 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:11779 3.9 e40/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 40'75, mlcod=40'73)", 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 121.042247996, 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.013535358000000001, 2024-09-16T13:19:18.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553492+0000", 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.626 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.553495+0000", 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560539+0000", 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0070444430000000001 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560565+0000", 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.6316e-05 2024-09-16T13:19:18.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560575+0000", 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0189e-05 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560646+0000", 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0845000000000004e-05 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.560680+0000", 2024-09-16T13:19:18.628 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3874000000000001e-05 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.566960+0000", 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062798790000000004 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:17.567028+0000", 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.7549e-05 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15138 3.4 e49/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 49'43, mlcod=42'41)", 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 110.785834155, 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044817629999999997, 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809906+0000", 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.809907+0000", 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3430000000000001e-06 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.631 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810936+0000", 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0010289400000000001 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810958+0000", 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1209999999999999e-05 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.810966+0000", 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1550000000000007e-06 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811051+0000", 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.5321999999999994e-05 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.811096+0000", 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4990000000000001e-05 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814274+0000", 2024-09-16T13:19:18.633 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0031783409999999999 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:27.814388+0000", 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011346200000000001 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:15325 3.4 e50/23 3:23b9d2fd:::rbd_data.112d492326e3.0000000000000000:head v 50'45, mlcod=42'42)", 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 110.173271048, 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0056730849999999996, 2024-09-16T13:19:18.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422469+0000", 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422471+0000", 2024-09-16T13:19:18.635 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.513e-06 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422490+0000", 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9253999999999999e-05 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422520+0000", 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9521000000000001e-05 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.636 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422531+0000", 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1236000000000001e-05 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422636+0000", 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000105327 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.422682+0000", 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5550999999999998e-05 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428095+0000", 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0054134140000000001 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:28.428142+0000", 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7268999999999998e-05 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.638 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28178 3.0 e69/23 3:00e4afe7:::rbd_data.112dab4f0555.000000000000000c:head v 69'178, mlcod=68'175)", 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 83.335164878, 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048665119999999999, 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260575+0000", 2024-09-16T13:19:18.639 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260576+0000", 2024-09-16T13:19:18.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.7999999999999993e-07 2024-09-16T13:19:18.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.640 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.640 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260594+0000", 2024-09-16T13:19:18.640 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7292999999999999e-05 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260627+0000", 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3102999999999998e-05 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260644+0000", 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7574000000000001e-05 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.641 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260852+0000", 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00020765599999999999 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.260921+0000", 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8776999999999996e-05 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265335+0000", 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0044137719999999998 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.642 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.265442+0000", 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107357 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:28749 3.9 e69/23 3:969cb6ee:::rbd_data.112d770f4ae2.0000000000000002:head v 69'236, mlcod=68'233)", 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 82.971037545000002, 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0062636810000000001, 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.643 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624703+0000", 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.644 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.624704+0000", 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.6130000000000001e-06 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627509+0000", 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0028051539999999998 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627529+0000", 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9502999999999999e-05 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627536+0000", 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9940000000000003e-06 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627584+0000", 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8204999999999997e-05 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.627607+0000", 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.2736999999999999e-05 2024-09-16T13:19:18.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630903+0000", 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.003295798 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:17:55.630966+0000", 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3676999999999994e-05 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.647 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:75420 3.4 e88/23 3:2605e3b8:::rbd_object_map.112d9326a5dd.0000000000000026:head v 88'165, mlcod=88'163)", 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 59.092308529, 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0055875079999999997, 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:18.648 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503432+0000", 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503433+0000", 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4169999999999999e-06 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.649 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503451+0000", 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7640999999999999e-05 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503476+0000", 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5599999999999999e-05 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503495+0000", 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.825e-05 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503611+0000", 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00011631799999999999 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.503644+0000", 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3476000000000003e-05 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.508954+0000", 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0053093799999999998 2024-09-16T13:19:18.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:19.509019+0000", 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5425999999999993e-05 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1195 3.4 e101/23 3:24bef2c8:::rbd_data.112da28a57db.0000000000000020:27 v 101'326, mlcod=99'322)", 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.222094902999999, 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005296243, 2024-09-16T13:19:18.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373645+0000", 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373646+0000", 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7e-07 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.373653+0000", 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4000000000000003e-06 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374730+0000", 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00107654 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374741+0000", 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1474999999999999e-05 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374829+0000", 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.7584999999999996e-05 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374854+0000", 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.5269000000000002e-05 2024-09-16T13:19:18.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378893+0000", 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0040386770000000001 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.378942+0000", 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.656 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.1.0:1197 3.4 e101/23 3:26d31b55:::rbd_data.112da28a57db.000000000000000c:27 v 101'328, mlcod=99'322)", 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.220977830000002, 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046376150000000003, 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374762+0000", 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374763+0000", 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5600000000000005e-07 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374784+0000", 2024-09-16T13:19:18.658 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0781999999999999e-05 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374794+0000", 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0558e-05 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.374800+0000", 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2780000000000001e-06 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.659 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375389+0000", 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00058970500000000005 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.375407+0000", 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7801000000000001e-05 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379351+0000", 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0039435030000000001 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.660 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.379400+0000", 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9332000000000001e-05 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:89341 3.4 e101/23 3:21ab973d:::rbd_data.112da28a57db.0000000000000002:head v 101'329, mlcod=101'326)", 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 41.098518036999998, 2024-09-16T13:19:18.661 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005014859, 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497222+0000", 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.662 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497223+0000", 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8599999999999998e-07 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497231+0000", 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6709999999999999e-06 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497245+0000", 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.489e-05 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.663 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.497253+0000", 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6199999999999999e-06 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498106+0000", 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00085257599999999996 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.498130+0000", 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4776000000000001e-05 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.664 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502154+0000", 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004023703 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:37.502237+0000", 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.2936999999999996e-05 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.665 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(osd.4.0:2718 3.9 e106/23 3:9702159d:::rbd_data.112d404789fd.000000000000001c:22 v 106'440, mlcod=105'437)", 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 36.174699893000003, 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0063154409999999998, 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.666 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421040+0000", 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421041+0000", 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4700000000000002e-07 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421055+0000", 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4579999999999999e-05 2024-09-16T13:19:18.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421080+0000", 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4859e-05 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421089+0000", 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.8950000000000004e-06 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421166+0000", 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6892999999999997e-05 2024-09-16T13:19:18.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.421197+0000", 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1325000000000002e-05 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427304+0000", 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0061062449999999997 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:42.427356+0000", 2024-09-16T13:19:18.669 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2197000000000002e-05 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:94609 3.9 e133/23 3:9536537b:::rbd_data.112d21cb8818.0000000000000003:head v 133'502, mlcod=132'500)", 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:09.204270+0000", 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.3914706330000008, 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.004587041, 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.670 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.671 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204270+0000", 2024-09-16T13:19:18.671 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.672 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.672 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204270+0000", 2024-09-16T13:19:18.672 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204270+0000", 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.0399999999999995e-07 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204304+0000", 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3590999999999997e-05 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204343+0000", 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8634999999999999e-05 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204358+0000", 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5566999999999999e-05 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204468+0000", 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109862 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.204526+0000", 2024-09-16T13:19:18.674 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7841999999999999e-05 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.208761+0000", 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0042354469999999998 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.208857+0000", 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.5393e-05 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:95238 3.4 e139/23 3:23de2965:::rbd_data.112d64676e7f.0000000000000007:head v 139'365, mlcod=136'362)", 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:15.246402+0000", 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.3493380569999998, 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0051633470000000004, 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246402+0000", 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246402+0000", 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246404+0000", 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.677 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246423+0000", 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9330999999999998e-05 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246461+0000", 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7128000000000001e-05 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246477+0000", 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5926e-05 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.678 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246584+0000", 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000107215 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246640+0000", 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6529e-05 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251525+0000", 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0048844149999999996 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.679 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251566+0000", 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0893999999999998e-05 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:95239 3.4 e139/23 3:249f6365:::rbd_data.112d64676e7f.0000000000000008:head v 139'366, mlcod=136'362)", 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:15.246826+0000", 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.3489145589999998, 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0047688360000000003, 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.680 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246826+0000", 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246826+0000", 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246827+0000", 2024-09-16T13:19:18.681 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:19:18.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.682 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.682 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246839+0000", 2024-09-16T13:19:18.682 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.1953999999999999e-05 2024-09-16T13:19:18.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.683 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.683 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246860+0000", 2024-09-16T13:19:18.683 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1474000000000001e-05 2024-09-16T13:19:18.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.246872+0000", 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.2109e-05 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.247634+0000", 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00076168299999999998 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.247679+0000", 2024-09-16T13:19:18.684 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.5404000000000001e-05 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251570+0000", 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0038906140000000001 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251595+0000", 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.4443999999999998e-05 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:95247 3.4 e139/23 3:248e1f52:::rbd_data.112d64676e7f.0000000000000036:head v 139'368, mlcod=136'362)", 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:15.250634+0000", 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.3451067669999999, 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005299967, 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250634+0000", 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250634+0000", 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250634+0000", 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.3799999999999996e-07 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250641+0000", 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.2130000000000002e-06 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250657+0000", 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5096000000000001e-05 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250664+0000", 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.6879999999999999e-06 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250810+0000", 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00014586699999999999 2024-09-16T13:19:18.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.250848+0000", 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8220999999999999e-05 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.255895+0000", 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.005047134 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.255933+0000", 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.8009999999999997e-05 2024-09-16T13:19:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_repop(client.4397.0:95250 3.0 e139/23 3:040d935c:::rbd_data.112d64676e7f.0000000000000051:head v 139'418, mlcod=136'415)", 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:15.251287+0000", 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 3.3444531770000001, 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0046599950000000001, 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:18.690 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251287+0000", 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251287+0000", 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251288+0000", 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.85e-07 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:18.691 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251294+0000", 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.3099999999999997e-06 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251308+0000", 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.3312000000000001e-05 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251315+0000", 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4170000000000003e-06 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.692 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251362+0000", 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.7203000000000002e-05 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.251378+0000", 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5899000000000001e-05 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "commit_sent", 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.255918+0000", 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.0045400600000000003 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:18.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:18.694 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:18.694 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:15.255947+0000", 2024-09-16T13:19:18.694 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9009e-05 2024-09-16T13:19:18.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:18.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:18.694 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:18.695 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6: {} 2024-09-16T13:19:18.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T13:19:18.696 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7: {} 2024-09-16T13:19:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T13:19:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T13:19:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T13:19:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T13:19:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T13:19:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T13:19:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T13:19:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T13:19:18.698 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:18.688+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:18.796 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-16T13:19:18.788+0000 7fa457913700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22264) UID: 0 2024-09-16T13:19:18.844 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:18.845 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:18.845 INFO:teuthology.orchestra.run.smithi098.stdout: "num_ops": 0 2024-09-16T13:19:18.845 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:18.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T13:19:18.897 INFO:tasks.ceph.osd.4.smithi151.stderr:2024-09-16T13:19:18.894+0000 7f67296ba700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 22276) UID: 0 2024-09-16T13:19:18.950 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [], 2024-09-16T13:19:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: "complaint_time": 30, 2024-09-16T13:19:18.951 INFO:teuthology.orchestra.run.smithi098.stdout: "num_blocked_ops": 0 2024-09-16T13:19:18.951 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:18.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T13:19:18.997 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-16T13:19:18.988+0000 7f5402146700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 22259) UID: 0 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout: "size": 20, 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 600, 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout: "ops": [ 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62302 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7839744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744822799000005, 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028011366999999999, 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.064 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62302 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315841+0000", 2024-09-16T13:19:19.065 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315842+0000", 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315851+0000", 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.815e-06 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315858+0000", 2024-09-16T13:19:19.066 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4479999999999999e-06 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315861+0000", 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2820000000000001e-06 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343757+0000", 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.027895413000000001 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343801+0000", 2024-09-16T13:19:19.067 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4718000000000001e-05 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343853+0000", 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.1347000000000002e-05 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62303 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7847936~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744798833000004, 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028109673000000002, 2024-09-16T13:19:19.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62303 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.069 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315865+0000", 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315866+0000", 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315868+0000", 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.3929999999999998e-06 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.070 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315874+0000", 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.4519999999999996e-06 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315877+0000", 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.3390000000000001e-06 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343877+0000", 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028000381000000001 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.071 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343920+0000", 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2286000000000001e-05 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.343975+0000", 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5373e-05 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62304 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7856128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744783663000007, 2024-09-16T13:19:19.072 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028254531999999999, 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62304 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.073 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315880+0000", 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315881+0000", 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7899999999999999e-07 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315882+0000", 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.621e-06 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.074 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315888+0000", 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8599999999999998e-06 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315891+0000", 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9179999999999998e-06 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344005+0000", 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028113863999999999 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.075 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344071+0000", 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5579000000000004e-05 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344135+0000", 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4311000000000001e-05 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62305 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744769239999997, 2024-09-16T13:19:19.076 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028428815, 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62305 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:19.077 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315895+0000", 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6100000000000002e-07 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.078 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.079 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315902+0000", 2024-09-16T13:19:19.079 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8279999999999997e-06 2024-09-16T13:19:19.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315909+0000", 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4699999999999999e-06 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315912+0000", 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4750000000000002e-06 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344175+0000", 2024-09-16T13:19:19.080 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028262879000000001 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344231+0000", 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5893000000000002e-05 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344324+0000", 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2909000000000001e-05 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62306 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7872512~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744747994999997, 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028537021999999999, 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62306 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:19.082 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315916+0000", 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7500000000000001e-07 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315918+0000", 2024-09-16T13:19:19.083 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.95e-06 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315924+0000", 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6389999999999997e-06 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315927+0000", 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0089999999999999e-06 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344346+0000", 2024-09-16T13:19:19.084 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028418589000000001 2024-09-16T13:19:19.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.085 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.085 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344395+0000", 2024-09-16T13:19:19.085 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9521000000000002e-05 2024-09-16T13:19:19.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.086 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.086 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344453+0000", 2024-09-16T13:19:19.086 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7939000000000002e-05 2024-09-16T13:19:19.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.086 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62308 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7880704~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744733425999996, 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028692415999999998, 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.087 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62308 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.088 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315931+0000", 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.53e-07 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315933+0000", 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7e-06 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315939+0000", 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0190000000000003e-06 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.089 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315946+0000", 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.832e-06 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344486+0000", 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028540052999999999 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344557+0000", 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.1210000000000004e-05 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.090 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344623+0000", 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6248999999999995e-05 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62331 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7888896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744714657000003, 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028812566000000001, 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.091 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62331 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315949+0000", 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315950+0000", 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315952+0000", 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.894e-06 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315957+0000", 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6520000000000003e-06 2024-09-16T13:19:19.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315961+0000", 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.162e-06 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344648+0000", 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028687275000000002 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344704+0000", 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5831000000000001e-05 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344762+0000", 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8354999999999999e-05 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62335 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7897088~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744699761999996, 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028964789000000001, 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.095 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.096 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.096 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62335 2024-09-16T13:19:19.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.096 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.096 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.096 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:19:19.096 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315964+0000", 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315965+0000", 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7000000000000005e-07 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315967+0000", 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.849e-06 2024-09-16T13:19:19.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315972+0000", 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.2109999999999997e-06 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315975+0000", 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0359999999999998e-06 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344798+0000", 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028822558000000002 2024-09-16T13:19:19.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344860+0000", 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2722e-05 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344929+0000", 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.8843000000000005e-05 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.099 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62338 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7905280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744685551000003, 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029089706, 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62338 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.100 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315979+0000", 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.46e-07 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315981+0000", 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.813e-06 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315986+0000", 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8849999999999998e-06 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315989+0000", 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.9129999999999999e-06 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.102 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.344954+0000", 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.028965270000000001 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345010+0000", 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6711e-05 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345068+0000", 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7768e-05 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.103 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62340 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7913472~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744672182000002, 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029254848, 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62340 2024-09-16T13:19:19.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.105 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:19.105 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.105 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.105 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315992+0000", 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5499999999999999e-07 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.315994+0000", 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.886e-06 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.106 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316000+0000", 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6999999999999996e-06 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316003+0000", 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.0900000000000001e-06 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345117+0000", 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029114068999999999 2024-09-16T13:19:19.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345182+0000", 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4709999999999995e-05 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345247+0000", 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.5037999999999996e-05 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.108 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62342 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7921664~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744657500000002, 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029394512000000001, 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.109 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62342 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316007+0000", 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.27e-07 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316009+0000", 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316015+0000", 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8640000000000001e-06 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.111 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316018+0000", 2024-09-16T13:19:19.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4630000000000001e-06 2024-09-16T13:19:19.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.112 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.112 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345309+0000", 2024-09-16T13:19:19.112 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029290495999999999 2024-09-16T13:19:19.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345346+0000", 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.7166999999999998e-05 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345401+0000", 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.5148999999999997e-05 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62343 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7929856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.113 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744642330999994, 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029516737000000001, 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62343 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:19.114 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316022+0000", 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.27e-07 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316024+0000", 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.8190000000000001e-06 2024-09-16T13:19:19.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316030+0000", 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316033+0000", 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2830000000000002e-06 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345430+0000", 2024-09-16T13:19:19.116 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029397666999999999 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345479+0000", 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.8927000000000001e-05 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345539+0000", 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9104000000000002e-05 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62346 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7938048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744627691000005, 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029636992000000001, 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62346 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.118 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316036+0000", 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316037+0000", 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.4999999999999998e-07 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.119 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316053+0000", 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.5809000000000001e-05 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316060+0000", 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.9e-06 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316063+0000", 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6049999999999998e-06 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345554+0000", 2024-09-16T13:19:19.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029491264 2024-09-16T13:19:19.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.121 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.121 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345615+0000", 2024-09-16T13:19:19.121 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0776000000000002e-05 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345673+0000", 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8187999999999999e-05 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62348 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7946240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:19:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744596963000006, 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029741212, 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62348 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316067+0000", 2024-09-16T13:19:19.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.124 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.124 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316068+0000", 2024-09-16T13:19:19.124 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.2800000000000002e-07 2024-09-16T13:19:19.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316070+0000", 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.968e-06 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316075+0000", 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.7139999999999998e-06 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316084+0000", 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.918e-06 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345701+0000", 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029617193999999999 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345748+0000", 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6144999999999999e-05 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345808+0000", 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0844999999999998e-05 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62350 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7954432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744575945999998, 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029847577, 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62350 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316088+0000", 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316089+0000", 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6800000000000001e-07 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316091+0000", 2024-09-16T13:19:19.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.1919999999999999e-06 2024-09-16T13:19:19.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.129 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.129 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316097+0000", 2024-09-16T13:19:19.129 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.9410000000000004e-06 2024-09-16T13:19:19.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316104+0000", 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 7.4819999999999997e-06 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345833+0000", 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029728471999999999 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345879+0000", 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.6461999999999997e-05 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345936+0000", 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6660000000000003e-05 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62352 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744555837999997, 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029964858, 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.131 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62352 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316108+0000", 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316109+0000", 2024-09-16T13:19:19.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.03e-07 2024-09-16T13:19:19.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.133 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.133 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316115+0000", 2024-09-16T13:19:19.133 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8880000000000002e-06 2024-09-16T13:19:19.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316121+0000", 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.0260000000000004e-06 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316124+0000", 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.5130000000000001e-06 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.345970+0000", 2024-09-16T13:19:19.134 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029845944999999999 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346010+0000", 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.0292e-05 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346073+0000", 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.2791000000000002e-05 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.135 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62356 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7970816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744535935000002, 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030210323000000001, 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62356 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.136 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316128+0000", 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316129+0000", 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6699999999999999e-07 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.137 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316131+0000", 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.0660000000000002e-06 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316136+0000", 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.8050000000000003e-06 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316140+0000", 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.2030000000000001e-06 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346115+0000", 2024-09-16T13:19:19.138 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.029975003 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346207+0000", 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 9.2378000000000002e-05 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346338+0000", 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013150099999999999 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62358 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744520722000004, 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030349426999999998, 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62358 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316143+0000", 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316144+0000", 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.6300000000000001e-07 2024-09-16T13:19:19.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316146+0000", 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9640000000000002e-06 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316152+0000", 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.4520000000000002e-06 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316155+0000", 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.1360000000000001e-06 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346362+0000", 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030206490999999999 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346411+0000", 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 4.9203999999999998e-05 2024-09-16T13:19:19.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346493+0000", 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.1817000000000006e-05 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:62359 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:21 [read 7987200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 65.744505188000005, 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030579776999999999, 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.144 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 62359 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:19.145 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316159+0000", 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 3.4299999999999999e-07 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316161+0000", 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.9089999999999998e-06 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316167+0000", 2024-09-16T13:19:19.146 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 5.6099999999999997e-06 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.316170+0000", 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 2.7930000000000002e-06 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346521+0000", 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.030351110000000001 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346607+0000", 2024-09-16T13:19:19.147 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 8.6075000000000005e-05 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:18:13.346739+0000", 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00013193699999999999 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: "description": "osd_op(client.4397.0:94689 3.18 3:1a954864:::rbd_object_map.112ddef65b5a.000000000000000b:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e133)", 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: "initiated_at": "2024-09-16T13:19:09.325538+0000", 2024-09-16T13:19:19.148 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 9.7351262730000006, 2024-09-16T13:19:19.149 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.74768336999999996, 2024-09-16T13:19:19.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type_data": { 2024-09-16T13:19:19.149 INFO:teuthology.orchestra.run.smithi098.stdout: "flag_point": "started", 2024-09-16T13:19:19.149 INFO:teuthology.orchestra.run.smithi098.stdout: "client_info": { 2024-09-16T13:19:19.149 INFO:teuthology.orchestra.run.smithi098.stdout: "client": "client.4397", 2024-09-16T13:19:19.149 INFO:teuthology.orchestra.run.smithi098.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tid": 94689 2024-09-16T13:19:19.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.149 INFO:teuthology.orchestra.run.smithi098.stdout: "events": [ 2024-09-16T13:19:19.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "initiated", 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.325538+0000", 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "header_read", 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.325538+0000", 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "throttled", 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.325539+0000", 2024-09-16T13:19:19.150 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4580000000000001e-06 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "all_read", 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.325546+0000", 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 6.6769999999999998e-06 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "dispatched", 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.325564+0000", 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.7855000000000001e-05 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.325574+0000", 2024-09-16T13:19:19.151 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.0491e-05 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.325684+0000", 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.000109487 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "waiting for peered", 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:09.325698+0000", 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 1.4158999999999999e-05 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "reached_pg", 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:10.072658+0000", 2024-09-16T13:19:19.152 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.74696014300000002 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "started", 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:10.072888+0000", 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00022999699999999999 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: "event": "done", 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: "time": "2024-09-16T13:19:10.073221+0000", 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: "duration": 0.00033310300000000001 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.153 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-16T13:19:19.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-16T13:19:19.154 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:19.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T13:19:19.155 INFO:tasks.ceph.osd.6.smithi151.stderr:2024-09-16T13:19:19.146+0000 7fa066d36700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 22282) UID: 0 2024-09-16T13:19:19.255 INFO:tasks.ceph.osd.3.smithi098.stderr:2024-09-16T13:19:19.248+0000 7f872e5df700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 22260) UID: 0 2024-09-16T13:19:19.261 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:19.261 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:19.261 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:19.261 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:19.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T13:19:19.273 INFO:teuthology.orchestra.run.smithi098.stdout:All operations completed A-OK! 2024-09-16T13:19:19.315 DEBUG:teuthology.parallel:result is None 2024-09-16T13:19:19.315 DEBUG:teuthology.run_tasks:Unwinding manager rbd_fsx 2024-09-16T13:19:19.324 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T13:19:19.334 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T13:19:19.364 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:19.364 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:19.365 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:19.365 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:19.365 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:19.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T13:19:19.474 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:19.474 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:19.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:19.474 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:61002 3.1b 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 8388608 write_size 8388608,write 2244980~59057] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 66.259706082999998, 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.060341961999999999, 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 61002 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209784+0000", 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.209787+0000", 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1870000000000001e-06 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249689+0000", 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039902523000000002 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249723+0000", 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3946999999999999e-05 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249735+0000", 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.249876+0000", 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085700000000001 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.250274+0000", 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039815 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269855+0000", 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019581003999999999 2024-09-16T13:19:19.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.269995+0000", 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013974500000000001 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:13.270126+0000", 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130949 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.479 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75320 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1114112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.014466357000003, 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048391086999999999, 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75320 2024-09-16T13:19:19.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455023+0000", 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455024+0000", 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8100000000000003e-07 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455026+0000", 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2129999999999998e-06 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455033+0000", 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8630000000000002e-06 2024-09-16T13:19:19.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455037+0000", 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.608e-06 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503178+0000", 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048141646000000003 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503284+0000", 2024-09-16T13:19:19.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105917 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503415+0000", 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130359 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75321 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 1441792~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.014449364999997, 2024-09-16T13:19:19.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048644984000000002, 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75321 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:19:19.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455040+0000", 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455041+0000", 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455043+0000", 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0480000000000001e-06 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455048+0000", 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455052+0000", 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000002e-06 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503461+0000", 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048409079000000001 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503580+0000", 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119234 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503685+0000", 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105006 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75322 3.3 3:c71b699a:::rbd_data.112d3c2065c6.000000000000002a:25 [read 2064384~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.014423968000003, 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048835044000000001, 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75322 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455066+0000", 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2599999999999998e-07 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455068+0000", 2024-09-16T13:19:19.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.092e-06 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455074+0000", 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.925e-06 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455078+0000", 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.799e-06 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503740+0000", 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048661514000000003 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503822+0000", 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2196999999999999e-05 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503901+0000", 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9091000000000006e-05 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75344 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 589824~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013920327999998, 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048412121000000002, 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75344 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455569+0000", 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455570+0000", 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455572+0000", 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2749999999999998e-06 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455578+0000", 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455582+0000", 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4789999999999998e-06 2024-09-16T13:19:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503719+0000", 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048137054999999998 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503819+0000", 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9938000000000004e-05 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.503982+0000", 2024-09-16T13:19:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016225500000000001 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75345 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 917504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013903661999997, 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048702503000000001, 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75345 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455586+0000", 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455587+0000", 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455589+0000", 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9870000000000002e-06 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455594+0000", 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6840000000000001e-06 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455598+0000", 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7009999999999998e-06 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504027+0000", 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048429382999999999 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504123+0000", 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5251e-05 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504289+0000", 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016599200000000001 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75346 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1212416~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013888305000002, 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048896186000000001, 2024-09-16T13:19:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75346 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455601+0000", 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455602+0000", 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2799999999999996e-07 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455604+0000", 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455609+0000", 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455615+0000", 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2020000000000003e-06 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504315+0000", 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048700565000000001 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504382+0000", 2024-09-16T13:19:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7088999999999994e-05 2024-09-16T13:19:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504498+0000", 2024-09-16T13:19:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115408 2024-09-16T13:19:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75347 3.1b 3:d80f9f11:::rbd_data.112d3c2065c6.000000000000003f:25 [read 1540096~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013869759000002, 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049121102999999999, 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75347 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455620+0000", 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7100000000000002e-07 2024-09-16T13:19:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455623+0000", 2024-09-16T13:19:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0590000000000001e-06 2024-09-16T13:19:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455629+0000", 2024-09-16T13:19:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.314e-06 2024-09-16T13:19:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455633+0000", 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9400000000000004e-06 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504546+0000", 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048912718000000001 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504623+0000", 2024-09-16T13:19:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7726999999999996e-05 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504741+0000", 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117874 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75353 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 98304~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013734014999997, 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048929180000000003, 2024-09-16T13:19:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75353 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455756+0000", 2024-09-16T13:19:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0800000000000004e-07 2024-09-16T13:19:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455758+0000", 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8789999999999999e-06 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455763+0000", 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2190000000000003e-06 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455767+0000", 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5880000000000002e-06 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504529+0000", 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048761566999999999 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504587+0000", 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8564999999999999e-05 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504685+0000", 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7754e-05 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75354 3.13 3:ce4df0d4:::rbd_data.112d3c2065c6.0000000000000041:25 [read 393216~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013698945000002, 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049086520000000002, 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75354 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455791+0000", 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1099999999999996e-07 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455794+0000", 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.508e-06 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455801+0000", 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5630000000000003e-06 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455805+0000", 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.019e-06 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504704+0000", 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.048898981000000001 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504759+0000", 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4330000000000003e-05 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504877+0000", 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118608 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75355 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013680313000002, 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118086999999998, 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75355 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455809+0000", 2024-09-16T13:19:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455810+0000", 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000002e-07 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455812+0000", 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2630000000000002e-06 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455818+0000", 2024-09-16T13:19:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455821+0000", 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5980000000000001e-06 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504914+0000", 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049092961999999997 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504928+0000", 2024-09-16T13:19:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3210000000000001e-05 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75356 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013664552000002, 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049129843999999999, 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75356 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455825+0000", 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455826+0000", 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0399999999999996e-07 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455828+0000", 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-06 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455835+0000", 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8410000000000002e-06 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455839+0000", 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2400000000000001e-06 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504947+0000", 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049108546000000003 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504955+0000", 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6109999999999996e-06 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75357 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013647288000001, 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049141246, 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75357 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455842+0000", 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455843+0000", 2024-09-16T13:19:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3299999999999998e-07 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455845+0000", 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999998e-06 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455852+0000", 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9489999999999999e-06 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455856+0000", 2024-09-16T13:19:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6059999999999999e-06 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504974+0000", 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049118385000000001 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.504984+0000", 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7880000000000001e-06 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75358 3.13 3.2b0fb273 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013630339000002, 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049149627000000001, 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75358 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455859+0000", 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455860+0000", 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455862+0000", 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9429999999999999e-06 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455867+0000", 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.395e-06 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455873+0000", 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0120000000000002e-06 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505002+0000", 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049128988999999998 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505009+0000", 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7950000000000002e-06 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75370 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013612586000001, 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049174830000000003, 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75370 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455877+0000", 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455878+0000", 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999995e-07 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455880+0000", 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9460000000000001e-06 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455885+0000", 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3650000000000003e-06 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455889+0000", 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6090000000000001e-06 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505040+0000", 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049151242999999997 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505052+0000", 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2055999999999999e-05 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75371 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013597449999999, 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049181864999999998, 2024-09-16T13:19:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75371 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455892+0000", 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455893+0000", 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000002e-07 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455895+0000", 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455900+0000", 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.536e-06 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455904+0000", 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505067+0000", 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049163224999999998 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505074+0000", 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0700000000000001e-06 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75372 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013582311999997, 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049214870000000001, 2024-09-16T13:19:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75372 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455907+0000", 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455908+0000", 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.89e-07 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455910+0000", 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9010000000000001e-06 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455915+0000", 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3519999999999997e-06 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455919+0000", 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.472e-06 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505109+0000", 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049190747 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505122+0000", 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3009e-05 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75373 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013567795, 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049224933999999998, 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75373 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:19.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455922+0000", 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455924+0000", 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9290000000000001e-06 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455930+0000", 2024-09-16T13:19:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4929999999999998e-06 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455936+0000", 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4459999999999998e-06 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505140+0000", 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049203510999999998 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505147+0000", 2024-09-16T13:19:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1670000000000002e-06 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75374 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013549716, 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049235427999999998, 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75374 2024-09-16T13:19:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455940+0000", 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4400000000000001e-07 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455946+0000", 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860000000000001e-06 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455953+0000", 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5119999999999997e-06 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455956+0000", 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.579e-06 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505166+0000", 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049209527000000003 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505175+0000", 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:75375 3.13 3.c6e8d353 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e88)", 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 60.013529435999999, 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049243375999999998, 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75375 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455960+0000", 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455961+0000", 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455963+0000", 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729999999999999e-06 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455970+0000", 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2520000000000002e-06 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.455975+0000", 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505198+0000", 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049222852999999997 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:19.505204+0000", 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.559 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:19.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T13:19:19.667 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:19.667 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:19.667 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:19.667 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:19.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T13:19:19.777 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:19.777 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:19.777 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:19.777 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:19.778 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:19.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T13:19:19.889 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:31819 3.1f 3:fc3d7dde:::rbd_data.112dac95762d.0000000000000052:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 79.063913188000001, 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28126801200000001, 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 31819 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822471+0000", 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822473+0000", 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1169999999999998e-06 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822484+0000", 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0947e-05 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822545+0000", 2024-09-16T13:19:19.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0414999999999997e-05 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822561+0000", 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6294999999999999e-05 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822663+0000", 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102407 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:19.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:00.822682+0000", 2024-09-16T13:19:19.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8749e-05 2024-09-16T13:19:19.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.100739+0000", 2024-09-16T13:19:19.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805706899999999 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101146+0000", 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040635699999999999 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.101645+0000", 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049913900000000001 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103268+0000", 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016229650000000001 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103361+0000", 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3108000000000004e-05 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103487+0000", 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126358 2024-09-16T13:19:19.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:01.103739+0000", 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000252086 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82800 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1130496~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:19:19.898 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027855092999999, 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050353660000000001, 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82800 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858529+0000", 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858530+0000", 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858534+0000", 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8229999999999997e-06 2024-09-16T13:19:19.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858547+0000", 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3302000000000001e-05 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858557+0000", 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.012e-05 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908637+0000", 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050079811000000002 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908731+0000", 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4085000000000005e-05 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908883+0000", 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000151676 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.902 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82801 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1212416~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027819198000003, 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050667642999999998, 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82801 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858565+0000", 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858566+0000", 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3499999999999995e-07 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858570+0000", 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8299999999999998e-06 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858590+0000", 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9831e-05 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858597+0000", 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6729999999999992e-06 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.908930+0000", 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050332492999999999 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909036+0000", 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010603999999999999 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909233+0000", 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019704100000000001 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82802 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027778902000001, 2024-09-16T13:19:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050929965000000001, 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82802 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:19.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858606+0000", 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858610+0000", 2024-09-16T13:19:19.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0820000000000001e-06 2024-09-16T13:19:19.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858655+0000", 2024-09-16T13:19:19.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4901000000000003e-05 2024-09-16T13:19:19.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858668+0000", 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2343e-05 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909278+0000", 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050610005 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909374+0000", 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6399999999999999e-05 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909535+0000", 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016137599999999999 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82803 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1474560~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027670051000001, 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051051886999999997, 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.912 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82803 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858714+0000", 2024-09-16T13:19:19.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858715+0000", 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999995e-07 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858719+0000", 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.732e-06 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858728+0000", 2024-09-16T13:19:19.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6730000000000004e-06 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858733+0000", 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.532e-06 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909572+0000", 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050839399 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909647+0000", 2024-09-16T13:19:19.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5073000000000004e-05 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909766+0000", 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118805 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82804 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1654784~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027645679000003, 2024-09-16T13:19:19.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051320085000000001, 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82804 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858739+0000", 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3399999999999998e-07 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858742+0000", 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4150000000000002e-06 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858748+0000", 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0090000000000004e-06 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858753+0000", 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1649999999999997e-06 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909812+0000", 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051058612000000003 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.909907+0000", 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5795e-05 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910059+0000", 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015135499999999999 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.920 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82805 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1736704~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027625763000003, 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051834002999999997, 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82805 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858759+0000", 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2200000000000004e-07 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858762+0000", 2024-09-16T13:19:19.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4959999999999999e-06 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858768+0000", 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8830000000000003e-06 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858771+0000", 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9319999999999998e-06 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910078+0000", 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051306696999999998 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910163+0000", 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5065e-05 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910593+0000", 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042940799999999999 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.924 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82806 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1916928~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027608981999997, 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052120149999999997, 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82806 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858775+0000", 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858776+0000", 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2600000000000002e-07 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858778+0000", 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3070000000000001e-06 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858784+0000", 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0639999999999999e-06 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858788+0000", 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8870000000000002e-06 2024-09-16T13:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910643+0000", 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051854355999999997 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910735+0000", 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2454000000000006e-05 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910896+0000", 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016055600000000001 2024-09-16T13:19:19.928 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82807 3.d 3:b03db375:::rbd_data.112da28a57db.0000000000000007:27 [read 1998848~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027589814000002, 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052345503000000002, 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.929 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.930 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82807 2024-09-16T13:19:19.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.930 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:19.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858795+0000", 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858797+0000", 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858804+0000", 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0079999999999996e-06 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858808+0000", 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.910935+0000", 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052126695000000001 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911010+0000", 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4764000000000004e-05 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911140+0000", 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013022500000000001 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.933 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82830 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 770048~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027571803999997, 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052616234999999997, 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82830 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858813+0000", 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858815+0000", 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3130000000000001e-06 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858821+0000", 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7200000000000003e-06 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858838+0000", 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6492000000000001e-05 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911178+0000", 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052340684999999998 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911272+0000", 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4106999999999999e-05 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911429+0000", 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156389 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82831 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027538016999998, 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052803142999999997, 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.938 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82831 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858846+0000", 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858848+0000", 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.209e-06 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858858+0000", 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0111e-05 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858870+0000", 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1976e-05 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858876+0000", 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5649999999999998e-06 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911449+0000", 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052573110999999999 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911521+0000", 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1209000000000002e-05 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911650+0000", 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012896199999999999 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82832 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1032192~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027500355999997, 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053031663999999999, 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82832 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858884+0000", 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858885+0000", 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6899999999999996e-07 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858889+0000", 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.087e-06 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858905+0000", 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5744999999999999e-05 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858909+0000", 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7269999999999998e-06 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911673+0000", 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052763697999999998 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911767+0000", 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4054000000000004e-05 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911916+0000", 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148584 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82833 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1114112~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.946 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027467252999998, 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053225354000000002, 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82833 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858917+0000", 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858918+0000", 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0869999999999999e-06 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858922+0000", 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.371e-06 2024-09-16T13:19:19.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858933+0000", 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1076e-05 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858940+0000", 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9840000000000004e-06 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.911944+0000", 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053004044 2024-09-16T13:19:19.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912021+0000", 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7189999999999998e-05 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912143+0000", 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012160200000000001 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.950 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82834 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1294336~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027437501000001, 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053469475000000002, 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82834 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858947+0000", 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858948+0000", 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858952+0000", 2024-09-16T13:19:19.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:19:19.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858970+0000", 2024-09-16T13:19:19.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.889e-05 2024-09-16T13:19:19.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858978+0000", 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0080000000000002e-06 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912172+0000", 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053193670999999998 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912268+0000", 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6100000000000005e-05 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912416+0000", 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014817500000000001 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82835 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1376256~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027397207, 2024-09-16T13:19:19.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053671286999999998, 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82835 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858987+0000", 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858988+0000", 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858992+0000", 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8539999999999999e-06 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.858999+0000", 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859003+0000", 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9099999999999998e-06 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912456+0000", 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053453374999999997 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912526+0000", 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9237000000000001e-05 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912658+0000", 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013284000000000001 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82836 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1556480~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027377360000003, 2024-09-16T13:19:19.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053937880000000001, 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82836 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859007+0000", 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859008+0000", 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5799999999999999e-07 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859010+0000", 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1950000000000002e-06 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859016+0000", 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8429999999999998e-06 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859019+0000", 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7639999999999999e-06 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912697+0000", 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053677322 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912796+0000", 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8991000000000001e-05 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912945+0000", 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149207 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82837 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1638400~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:19:19.963 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027361337000002, 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054154068999999999, 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82837 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859023+0000", 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859024+0000", 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1600000000000001e-07 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859026+0000", 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-06 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859031+0000", 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6690000000000003e-06 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859035+0000", 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6239999999999999e-06 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.912981+0000", 2024-09-16T13:19:19.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053946409000000001 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913054+0000", 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3052999999999993e-05 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913177+0000", 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122803 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82838 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1818624~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027345898999997, 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054416554999999998, 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82838 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:19.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859039+0000", 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859041+0000", 2024-09-16T13:19:19.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2139999999999999e-06 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859047+0000", 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6359999999999999e-06 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859051+0000", 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6519999999999999e-06 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913215+0000", 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054164344000000003 2024-09-16T13:19:19.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913307+0000", 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2546999999999993e-05 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913455+0000", 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147639 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.971 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82839 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 1900544~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027330237000001, 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054634963000000002, 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82839 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:19:19.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859054+0000", 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859055+0000", 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859057+0000", 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3910000000000001e-06 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859066+0000", 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5150000000000006e-06 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859072+0000", 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5810000000000002e-06 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913493+0000", 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054420751000000003 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913563+0000", 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9938999999999996e-05 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913689+0000", 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012626299999999999 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:82840 3.d 3:b2772ba2:::rbd_data.112da28a57db.000000000000000a:27 [read 2080768~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.027303201000002, 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054884993999999999, 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82840 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.976 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859081+0000", 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859082+0000", 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5799999999999998e-07 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859086+0000", 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7510000000000002e-06 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859097+0000", 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1085e-05 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.859104+0000", 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-06 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913718+0000", 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.054614085 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913814+0000", 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5514999999999996e-05 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:23.913966+0000", 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015253 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.980 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:19.980 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:19.980 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:19.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T13:19:19.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T13:19:20.086 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:20.086 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:20.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T13:19:20.195 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:20.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:20.195 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:20.195 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:20.195 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:20.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T13:19:20.302 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35848 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13172736~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417604573000006, 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050908163999999999, 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.303 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35848 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882010+0000", 2024-09-16T13:19:20.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882011+0000", 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882013+0000", 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4609999999999998e-06 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882019+0000", 2024-09-16T13:19:20.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2979999999999997e-06 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882024+0000", 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932797+0000", 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050772324000000001 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932852+0000", 2024-09-16T13:19:20.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5551999999999999e-05 2024-09-16T13:19:20.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932918+0000", 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6058999999999999e-05 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35849 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 13631488~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417585738, 2024-09-16T13:19:20.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051084381999999998, 2024-09-16T13:19:20.309 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.309 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.309 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35849 2024-09-16T13:19:20.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.309 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882029+0000", 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.46e-07 2024-09-16T13:19:20.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882032+0000", 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2290000000000002e-06 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882040+0000", 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882043+0000", 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000004e-06 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.932953+0000", 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050909821000000001 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933027+0000", 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4207000000000003e-05 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933113+0000", 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5805999999999998e-05 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35850 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14352384~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417566598999997, 2024-09-16T13:19:20.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051185040000000001, 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35850 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882048+0000", 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882049+0000", 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4900000000000001e-07 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882051+0000", 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.52e-06 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882057+0000", 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0610000000000001e-06 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882061+0000", 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9049999999999999e-06 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933130+0000", 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051069080000000003 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933173+0000", 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3016999999999998e-05 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933233+0000", 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0007999999999999e-05 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35851 3.a 3:5254d62e:::rbd_data.112d94eb720f.0000000000000002:20 [sparse-read 14811136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.317 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417549541, 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051370375000000003, 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35851 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:20.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882065+0000", 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882066+0000", 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6699999999999999e-07 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882068+0000", 2024-09-16T13:19:20.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1280000000000002e-06 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882074+0000", 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882078+0000", 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933274+0000", 2024-09-16T13:19:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051196341999999999 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933352+0000", 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8177999999999996e-05 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933435+0000", 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3381999999999996e-05 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35877 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 393216~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417532180999999, 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051687604999999998, 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35877 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882082+0000", 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882083+0000", 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882086+0000", 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882092+0000", 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3459999999999999e-06 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882096+0000", 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6660000000000001e-06 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933449+0000", 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051353453 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933494+0000", 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933770+0000", 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027599699999999999 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35878 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1114112~65536] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417514471999993, 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052102203, 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35878 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882100+0000", 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882101+0000", 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882103+0000", 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3180000000000001e-06 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882110+0000", 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.618e-06 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882114+0000", 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7450000000000001e-06 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933803+0000", 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051689582999999997 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.933918+0000", 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114782 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934202+0000", 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028446199999999997 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35879 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 1572864~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417497173000001, 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052211199999999999, 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35879 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882117+0000", 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882118+0000", 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.15e-07 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882129+0000", 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1496e-05 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882136+0000", 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7730000000000002e-06 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882146+0000", 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5799999999999998e-06 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934219+0000", 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052072887999999998 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934270+0000", 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1616000000000002e-05 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934329+0000", 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8431999999999999e-05 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.333 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35880 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2293760~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417462075000003, 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052388826999999999, 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35880 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.334 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882153+0000", 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.75e-07 2024-09-16T13:19:20.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882160+0000", 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1169999999999998e-06 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882167+0000", 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882171+0000", 2024-09-16T13:19:20.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6950000000000002e-06 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934374+0000", 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052202905000000001 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934449+0000", 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5123999999999994e-05 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934541+0000", 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2225999999999994e-05 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35881 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 2752512~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417438919999995, 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052480549000000001, 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.338 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35881 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882176+0000", 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4299999999999998e-07 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882179+0000", 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4569999999999999e-06 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882185+0000", 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6479999999999997e-06 2024-09-16T13:19:20.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882189+0000", 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8449999999999996e-06 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934558+0000", 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052368884999999997 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934603+0000", 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4750999999999999e-05 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934656+0000", 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3520000000000003e-05 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35882 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3473408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417397901000001, 2024-09-16T13:19:20.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0526574, 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35882 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882217+0000", 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882218+0000", 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882220+0000", 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7590000000000002e-06 2024-09-16T13:19:20.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882229+0000", 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.653e-06 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882235+0000", 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2330000000000003e-06 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934696+0000", 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052460397999999998 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934769+0000", 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3893000000000006e-05 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934874+0000", 2024-09-16T13:19:20.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104684 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35883 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 3932160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417375000999996, 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052754473000000003, 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.347 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35883 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882240+0000", 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882243+0000", 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4449999999999999e-06 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882249+0000", 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882260+0000", 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0698e-05 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934894+0000", 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052633895999999999 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934937+0000", 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3297000000000002e-05 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.934994+0000", 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7082e-05 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35884 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 4653056~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417349951999995, 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052926316000000001, 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.351 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35884 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882265+0000", 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9999999999999998e-07 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882273+0000", 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7519999999999996e-06 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882280+0000", 2024-09-16T13:19:20.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7649999999999996e-06 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882284+0000", 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9570000000000004e-06 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935036+0000", 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052752497000000002 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935108+0000", 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1452999999999995e-05 2024-09-16T13:19:20.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935191+0000", 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3492000000000006e-05 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35885 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5111808~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417324730999994, 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053012825999999999, 2024-09-16T13:19:20.355 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35885 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882290+0000", 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9299999999999998e-07 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882293+0000", 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5220000000000002e-06 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882300+0000", 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6610000000000003e-06 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882303+0000", 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.766e-06 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935205+0000", 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052901568000000003 2024-09-16T13:19:20.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935249+0000", 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3693000000000003e-05 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935303+0000", 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4122999999999997e-05 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.359 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35886 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 5832704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417306366000005, 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053204567000000001, 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35886 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882308+0000", 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882544+0000", 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000235393 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882548+0000", 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882559+0000", 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0883999999999999e-05 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882565+0000", 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935340+0000", 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052774561999999997 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935424+0000", 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4400999999999994e-05 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935513+0000", 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8564000000000001e-05 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.363 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35887 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 6291456~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417043229000001, 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053051908000000002, 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35887 2024-09-16T13:19:20.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882571+0000", 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882572+0000", 2024-09-16T13:19:20.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2300000000000001e-07 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882574+0000", 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.481e-06 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882582+0000", 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1069999999999999e-06 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882586+0000", 2024-09-16T13:19:20.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.07e-06 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935527+0000", 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052941569000000001 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935569+0000", 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1616000000000003e-05 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935623+0000", 2024-09-16T13:19:20.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4542000000000001e-05 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35888 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7012352~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.417018221000006, 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053248442, 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.368 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35888 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882596+0000", 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882597+0000", 2024-09-16T13:19:20.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5600000000000004e-07 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882600+0000", 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882607+0000", 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882619+0000", 2024-09-16T13:19:20.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935675+0000", 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053055395999999998 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935752+0000", 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7577999999999995e-05 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935845+0000", 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2585999999999998e-05 2024-09-16T13:19:20.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35889 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 7471104~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.416989889000007, 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053373167999999999, 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35889 2024-09-16T13:19:20.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882625+0000", 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882885+0000", 2024-09-16T13:19:20.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000260525 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882891+0000", 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882901+0000", 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0647e-05 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882907+0000", 2024-09-16T13:19:20.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9529999999999996e-06 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935890+0000", 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052982614999999997 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935938+0000", 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8618000000000001e-05 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.935998+0000", 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524999999999997e-05 2024-09-16T13:19:20.375 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35890 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8192000~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.416701191000001, 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053289387000000001, 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35890 2024-09-16T13:19:20.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882913+0000", 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882914+0000", 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882917+0000", 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3149999999999999e-06 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882923+0000", 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882929+0000", 2024-09-16T13:19:20.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9819999999999996e-06 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936041+0000", 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053112213999999998 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936118+0000", 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6908000000000003e-05 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936203+0000", 2024-09-16T13:19:20.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.488e-05 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35891 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 8650752~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.416681080999993, 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053389435999999998, 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35891 2024-09-16T13:19:20.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.882934+0000", 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886851+0000", 2024-09-16T13:19:20.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039169599999999997 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886856+0000", 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886871+0000", 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4888e-05 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.886879+0000", 2024-09-16T13:19:20.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.627e-06 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936220+0000", 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049341599 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936264+0000", 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3884999999999997e-05 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.936323+0000", 2024-09-16T13:19:20.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860999999999998e-05 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:35901 3.a 3:56756ad3:::rbd_data.112d94eb720f.0000000000000004:20 [sparse-read 14548992~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e78)", 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.412480157000005, 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050829543999999997, 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 35901 2024-09-16T13:19:20.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887134+0000", 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887135+0000", 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887138+0000", 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6539999999999999e-06 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887146+0000", 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8529999999999992e-06 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.887150+0000", 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3769999999999998e-06 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937868+0000", 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050717738999999998 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937910+0000", 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2215999999999997e-05 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:05.937964+0000", 2024-09-16T13:19:20.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3944999999999999e-05 2024-09-16T13:19:20.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.388 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.388 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.388 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:20.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T13:19:20.494 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:20.494 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:20.495 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-16T13:19:20.495 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:20.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T13:19:20.608 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:20.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-16T13:19:20.608 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-16T13:19:20.608 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-16T13:19:20.608 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:20.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T13:19:20.720 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-16T13:19:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-16T13:19:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-16T13:19:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-16T13:19:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90616 3.15 3:ae74f048:::rbd_data.112d420e934f.0000000000000002:head [delete] snapc 21=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.143764646000001, 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.870938027, 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90616 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570778+0000", 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570781+0000", 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.88e-06 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570792+0000", 2024-09-16T13:19:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0567000000000001e-05 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570816+0000", 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4219999999999999e-05 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570850+0000", 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3779999999999998e-05 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570946+0000", 2024-09-16T13:19:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6294999999999999e-05 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570958+0000", 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1661999999999999e-05 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438058+0000", 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86709994199999996 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438302+0000", 2024-09-16T13:19:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024435100000000002 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.438940+0000", 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063796000000000002 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441545+0000", 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026055509999999998 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441576+0000", 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0247e-05 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441622+0000", 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6047999999999997e-05 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.441716+0000", 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4524000000000005e-05 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90621 3.e 3.2a047d2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.143663406000002, 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.860052445, 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90621 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570880+0000", 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570881+0000", 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1540000000000001e-06 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570885+0000", 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7230000000000004e-06 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570901+0000", 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5756999999999998e-05 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.570908+0000", 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2960000000000001e-06 2024-09-16T13:19:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571040+0000", 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131803 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571069+0000", 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8694e-05 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430917+0000", 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85984788899999998 2024-09-16T13:19:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430932+0000", 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5129e-05 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90624 3.e 3.2071842e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.143080142000002, 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85950820299999997, 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90624 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571463+0000", 2024-09-16T13:19:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571465+0000", 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1799999999999999e-06 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571490+0000", 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5320000000000002e-05 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571524+0000", 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4190000000000003e-05 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571543+0000", 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8674000000000001e-05 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571647+0000", 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103803 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:43.571669+0000", 2024-09-16T13:19:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2354e-05 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430962+0000", 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85929225799999998 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.430971+0000", 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4239999999999999e-06 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2775 3.1b e108/107 3:dfdee96e:::rbd_data.112d338bfe8e.0000000000000000:23 v 108'605, mlcod=108'601)", 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.152369907000001, 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069960880000000001, 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562173+0000", 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562174+0000", 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7000000000000004e-07 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562181+0000", 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2459999999999997e-06 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562194+0000", 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2945000000000001e-05 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562200+0000", 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3679999999999998e-06 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562237+0000", 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7039999999999998e-05 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.562262+0000", 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4443999999999998e-05 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569096+0000", 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068345039999999999 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:44.569169+0000", 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2770999999999998e-05 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90825 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [watch reconnect cookie 140083147222288 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 34.263529192, 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0248692960000001, 2024-09-16T13:19:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90825 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451014+0000", 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451018+0000", 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8569999999999997e-06 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451032+0000", 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4586999999999999e-05 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451078+0000", 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6180000000000002e-05 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451101+0000", 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2773e-05 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451261+0000", 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016003600000000001 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.451276+0000", 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4387e-05 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459307+0000", 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080310969999999992 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459318+0000", 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1136e-05 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471312+0000", 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.011993798 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.471508+0000", 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019619700000000001 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.475883+0000", 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004375248 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.4.0:2793 3.1b e110/107 3:deba7c87:::rbd_data.112d94eb720f.0000000000000000:20 v 110'613, mlcod=108'609)", 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 34.258366371000001, 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070642279999999997, 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456177+0000", 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456178+0000", 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.517e-06 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456230+0000", 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2323e-05 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456251+0000", 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0619999999999999e-05 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456269+0000", 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7883000000000001e-05 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456632+0000", 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036355300000000002 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456686+0000", 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3454e-05 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463210+0000", 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065238550000000003 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.463241+0000", 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1022999999999998e-05 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:90826 3.e 3:70b1ef6c:::rbd_header.112d94eb720f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 34.258302114000003, 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.023049584, 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90826 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456241+0000", 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456244+0000", 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3000000000000002e-06 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456261+0000", 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7036000000000001e-05 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456294+0000", 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3059999999999999e-05 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456313+0000", 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8876999999999999e-05 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456444+0000", 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013131300000000001 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting_for_map not empty", 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.456462+0000", 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7920000000000001e-05 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459343+0000", 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028811539999999999 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:46.459351+0000", 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5870000000000004e-06 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472631+0000", 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0132794970000001 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472737+0000", 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106496 2024-09-16T13:19:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.473418+0000", 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068136 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.474875+0000", 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001456564 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479175+0000", 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042996450000000004 2024-09-16T13:19:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479238+0000", 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3610999999999998e-05 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.479290+0000", 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2163999999999997e-05 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1617 3.12 e111/110 3:4dcf0fd3:::rbd_data.112d404789fd.0000000000000011:22 v 111'330, mlcod=0'0)", 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 33.248572291999999, 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060955660000000002, 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465971+0000", 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465972+0000", 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2999999999999999e-07 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465984+0000", 2024-09-16T13:19:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2133999999999999e-05 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.465998+0000", 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4184e-05 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466009+0000", 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1371e-05 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467605+0000", 2024-09-16T13:19:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015953129999999999 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.467632+0000", 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7447e-05 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472017+0000", 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043852589999999999 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472066+0000", 2024-09-16T13:19:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8928000000000003e-05 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1618 3.12 e111/110 3:4bac8958:::rbd_data.112d404789fd.0000000000000008:22 v 111'332, mlcod=0'0)", 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 33.247975636, 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00616418, 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466567+0000", 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466568+0000", 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6000000000000003e-07 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466575+0000", 2024-09-16T13:19:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8809999999999998e-06 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466899+0000", 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032458399999999999 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.466912+0000", 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2767e-05 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469208+0000", 2024-09-16T13:19:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022958369999999998 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.469231+0000", 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2578e-05 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472690+0000", 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003459443 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.472731+0000", 2024-09-16T13:19:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1430000000000001e-05 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.0.0:1622 3.12 e111/110 3:498ca772:::rbd_data.112d12abb20f.0000000000000017:28 v 111'340, mlcod=111'334)", 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 33.22504842, 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074768710000000004, 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489495+0000", 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8599999999999997e-07 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489503+0000", 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8720000000000002e-06 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489515+0000", 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2191000000000001e-05 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489524+0000", 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6170000000000007e-06 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489592+0000", 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7670999999999996e-05 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.489614+0000", 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2112000000000001e-05 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496931+0000", 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073173739999999998 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:47.496971+0000", 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0247999999999998e-05 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:91826 3.11 3:8c261a44:::rbd_data.112d770f4ae2.000000000000000e:head [delete] snapc 1c=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.067971533000001, 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077560520000000003, 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 91826 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646571+0000", 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646576+0000", 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2699999999999998e-06 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646592+0000", 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6439999999999998e-05 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646637+0000", 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5120000000000002e-05 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646657+0000", 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9737000000000001e-05 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646782+0000", 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012466699999999999 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.646790+0000", 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6680000000000007e-06 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651149+0000", 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043590959999999998 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651538+0000", 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000389071 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.651714+0000", 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017587200000000001 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654242+0000", 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025272659999999998 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654249+0000", 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.61e-06 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654294+0000", 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4644000000000001e-05 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:51.654327+0000", 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3590999999999997e-05 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92599 3.11 3:8b1d31ff:::rbd_data.112d3e228c72.0000000000000004:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.075171101999999, 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070407359999999997, 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92599 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639372+0000", 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639375+0000", 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9399999999999998e-06 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639386+0000", 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1606999999999999e-05 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639426+0000", 2024-09-16T13:19:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9329000000000002e-05 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639439+0000", 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.348e-05 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639642+0000", 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000202976 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639865+0000", 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022277500000000001 2024-09-16T13:19:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640355+0000", 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049025000000000002 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.644293+0000", 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003937739 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646196+0000", 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019030410000000001 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646332+0000", 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013587899999999999 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646413+0000", 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.072e-05 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92601 3.13 3:cbdfcc30:::rbd_data.112d3e228c72.0000000000000006:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.075069461999998, 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069350899999999997, 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92601 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639473+0000", 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639475+0000", 2024-09-16T13:19:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.59e-06 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639482+0000", 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.46e-06 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639568+0000", 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6064999999999997e-05 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639591+0000", 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3564e-05 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.639945+0000", 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035337399999999999 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640456+0000", 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051193699999999996 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.640736+0000", 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027973199999999998 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.643326+0000", 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025895129999999999 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646146+0000", 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028207359999999999 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646277+0000", 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013046999999999999 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.646409+0000", 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013164900000000001 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:92605 3.11 3:8caec98f:::rbd_data.112d3e228c72.000000000000000a:head [delete] snapc 18=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-16T13:19:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.072634691000001, 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067373449999999996, 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92605 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641908+0000", 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641911+0000", 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-06 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641925+0000", 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.403e-05 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641959+0000", 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3448999999999998e-05 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.641976+0000", 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7397e-05 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.642102+0000", 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012618899999999999 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.642511+0000", 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040815799999999999 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.643089+0000", 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057817299999999997 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.644631+0000", 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015421969999999999 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648481+0000", 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038498930000000001 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648543+0000", 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2477000000000005e-05 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:18:55.648646+0000", 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010228199999999999 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:93417 3.e 3:74f3d54d:::rbd_data.112d30ff5030.000000000000000b:head [delete] snapc 13=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 20.617430649999999, 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077949680000000002, 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 93417 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097112+0000", 2024-09-16T13:19:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097116+0000", 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3859999999999999e-06 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097141+0000", 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5732e-05 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097191+0000", 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9854000000000001e-05 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097211+0000", 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0046000000000001e-05 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097304+0000", 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2318999999999995e-05 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.097311+0000", 2024-09-16T13:19:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4089999999999997e-06 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.101746+0000", 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044344780000000004 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.102087+0000", 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034153299999999998 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.102361+0000", 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027436799999999999 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104058+0000", 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001696431 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104805+0000", 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0007473 2024-09-16T13:19:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104848+0000", 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3112999999999999e-05 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:00.104907+0000", 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8999000000000002e-05 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:94264 3.13 3:ccbe4cb7:::rbd_id.image_client.0-clone10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:05.111021+0000", 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.603521859000001, 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068979260000000004, 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 94264 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111021+0000", 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111021+0000", 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111024+0000", 2024-09-16T13:19:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9510000000000002e-06 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111036+0000", 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2046e-05 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111071+0000", 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4898999999999999e-05 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111090+0000", 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9196999999999999e-05 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111231+0000", 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014067600000000001 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.111243+0000", 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2174000000000001e-05 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.115107+0000", 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038635219999999999 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.115225+0000", 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011879099999999999 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.115395+0000", 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016947899999999999 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117212+0000", 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001817319 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117788+0000", 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057622499999999998 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117862+0000", 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3218000000000003e-05 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:05.117919+0000", 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7429000000000002e-05 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:94516 3.13 3:ceb7d054:::rbd_header.112d21cb8818:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:09.057925+0000", 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 11.656617704, 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075387060000000001, 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.805 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 94516 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057925+0000", 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057925+0000", 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057928+0000", 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.526e-06 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057939+0000", 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1283e-05 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057961+0000", 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1962000000000001e-05 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.057985+0000", 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3867999999999999e-05 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.058984+0000", 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00099875000000000007 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.059268+0000", 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028392199999999998 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.060097+0000", 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00082984400000000005 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.064948+0000", 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048506900000000004 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.065340+0000", 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039161300000000002 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.065389+0000", 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9063000000000001e-05 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:09.065464+0000", 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5185000000000006e-05 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.810 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(osd.5.0:2212 3.16 e135/107 3:6d453e39:::rbd_data.112ddef65b5a.000000000000006b:b v 135'402, mlcod=120'397)", 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:11.068462+0000", 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.6460809300000001, 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005881346, 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068462+0000", 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068462+0000", 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068463+0000", 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.178e-06 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068474+0000", 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0382e-05 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068492+0000", 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8831999999999999e-05 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.068502+0000", 2024-09-16T13:19:20.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0066e-05 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.069980+0000", 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001477892 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.070006+0000", 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5978000000000001e-05 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.074244+0000", 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042378729999999996 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:11.074343+0000", 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9145e-05 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:95244 3.f 3:f2e9d83f:::rbd_data.112d64676e7f.000000000000000d:head [delete] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:15.244211+0000", 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.4703320629999999, 2024-09-16T13:19:20.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0097561130000000003, 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 95244 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.244211+0000", 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.244211+0000", 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.244214+0000", 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0369999999999999e-06 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.244228+0000", 2024-09-16T13:19:20.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3638999999999999e-05 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.244268+0000", 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0469999999999997e-05 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.244283+0000", 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5183e-05 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.244426+0000", 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142829 2024-09-16T13:19:20.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.244435+0000", 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8799999999999997e-06 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.249657+0000", 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005221598 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.250137+0000", 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00048075 2024-09-16T13:19:20.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4", 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.250383+0000", 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024584299999999998 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.252702+0000", 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023184099999999999 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.253867+0000", 2024-09-16T13:19:20.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011656279999999999 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.253923+0000", 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5702999999999998e-05 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:15.253967+0000", 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4143000000000001e-05 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.821 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.4397.0:95934 3.15 3:ae0349ea:::rbd_data.112de98401a1.0000000000000003:head [delete] snapc 4=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-16T13:19:19.240485+0000", 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.4740579300000001, 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057669119999999999, 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.4397", 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.98:0/397743624", 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 95934 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.240485+0000", 2024-09-16T13:19:20.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.240485+0000", 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.240486+0000", 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.240494+0000", 2024-09-16T13:19:20.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6680000000000001e-06 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.240513+0000", 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9020000000000001e-05 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.240524+0000", 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1022000000000001e-05 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.240699+0000", 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000174878 2024-09-16T13:19:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.241720+0000", 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001021311 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3", 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.241988+0000", 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000267996 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.245627+0000", 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036383349999999999 2024-09-16T13:19:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.246152+0000", 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052546699999999995 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.246207+0000", 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4608999999999998e-05 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-16T13:19:19.246252+0000", 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5277e-05 2024-09-16T13:19:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-16T13:19:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-16T13:19:20.827 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-16T13:19:21.157 INFO:teuthology.orchestra.run.smithi098.stderr:noscrub is unset 2024-09-16T13:19:21.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T13:19:22.181 INFO:teuthology.orchestra.run.smithi098.stderr:nodeep-scrub is unset 2024-09-16T13:19:23.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T13:19:23.241 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a: {} 2024-09-16T13:19:23.256 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b: {} 2024-09-16T13:19:23.273 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c: {} 2024-09-16T13:19:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:19:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:19:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T13:19:23.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T13:19:24.177 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T13:19:24.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-16T13:19:25.184 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T13:19:25.194 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T13:19:26.203 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T13:19:26.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T13:19:27.214 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T13:19:27.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T13:19:27.984 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T13:19:27.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T13:19:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T13:19:29.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T13:19:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T13:19:30.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-16T13:19:31.255 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T13:19:31.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T13:19:32.279 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T13:19:32.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-16T13:19:33.279 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T13:19:33.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T13:19:34.296 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T13:19:34.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-16T13:19:35.301 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T13:19:35.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T13:19:36.311 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T13:19:36.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-16T13:19:37.330 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T13:19:37.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T13:19:38.346 INFO:teuthology.orchestra.run.smithi098.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T13:19:38.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-16T13:19:39.347 INFO:teuthology.orchestra.run.smithi098.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T13:19:39.362 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T13:19:39.362 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T13:19:39.362 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T13:19:39.362 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T13:19:39.362 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T13:19:39.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:19:39.646 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:39.647 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":162,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:19:39.331549+0000","last_up_change":"2024-09-16T13:18:43.413345+0000","last_in_change":"2024-09-16T13:17:42.548353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:16:33.497561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:16:40.994538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-16T13:16:50.759705+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":81,"snap_epoch":143,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":22271},{"type":"v1","addr":"172.21.15.98:6809","nonce":22271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":22271},{"type":"v1","addr":"172.21.15.98:6811","nonce":22271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":22271},{"type":"v1","addr":"172.21.15.98:6817","nonce":22271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":22271},{"type":"v1","addr":"172.21.15.98:6814","nonce":22271}]},"public_addr":"172.21.15.98:6809/22271","cluster_addr":"172.21.15.98:6811/22271","heartbeat_back_addr":"172.21.15.98:6817/22271","heartbeat_front_addr":"172.21.15.98:6814/22271","state":["exists","up"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":22272},{"type":"v1","addr":"172.21.15.98:6815","nonce":22272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":22272},{"type":"v1","addr":"172.21.15.98:6819","nonce":22272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":22272},{"type":"v1","addr":"172.21.15.98:6823","nonce":22272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":22272},{"type":"v1","addr":"172.21.15.98:6821","nonce":22272}]},"public_addr":"172.21.15.98:6815/22272","cluster_addr":"172.21.15.98:6819/22272","heartbeat_back_addr":"172.21.15.98:6823/22272","heartbeat_front_addr":"172.21.15.98:6821/22272","state":["exists","up"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":22273},{"type":"v1","addr":"172.21.15.98:6801","nonce":22273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":22273},{"type":"v1","addr":"172.21.15.98:6803","nonce":22273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":22273},{"type":"v1","addr":"172.21.15.98:6807","nonce":22273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":22273},{"type":"v1","addr":"172.21.15.98:6805","nonce":22273}]},"public_addr":"172.21.15.98:6801/22273","cluster_addr":"172.21.15.98:6803/22273","heartbeat_back_addr":"172.21.15.98:6807/22273","heartbeat_front_addr":"172.21.15.98:6805/22273","state":["exists","up"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":22270},{"type":"v1","addr":"172.21.15.98:6825","nonce":22270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":22270},{"type":"v1","addr":"172.21.15.98:6827","nonce":22270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":22270},{"type":"v1","addr":"172.21.15.98:6831","nonce":22270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":22270},{"type":"v1","addr":"172.21.15.98:6829","nonce":22270}]},"public_addr":"172.21.15.98:6825/22270","cluster_addr":"172.21.15.98:6827/22270","heartbeat_back_addr":"172.21.15.98:6831/22270","heartbeat_front_addr":"172.21.15.98:6829/22270","state":["exists","up"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":22286},{"type":"v1","addr":"172.21.15.151:6825","nonce":22286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":22286},{"type":"v1","addr":"172.21.15.151:6827","nonce":22286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":22286},{"type":"v1","addr":"172.21.15.151:6831","nonce":22286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":22286},{"type":"v1","addr":"172.21.15.151:6829","nonce":22286}]},"public_addr":"172.21.15.151:6825/22286","cluster_addr":"172.21.15.151:6827/22286","heartbeat_back_addr":"172.21.15.151:6831/22286","heartbeat_front_addr":"172.21.15.151:6829/22286","state":["exists","up"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":22285},{"type":"v1","addr":"172.21.15.151:6809","nonce":22285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":22285},{"type":"v1","addr":"172.21.15.151:6811","nonce":22285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":22285},{"type":"v1","addr":"172.21.15.151:6815","nonce":22285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":22285},{"type":"v1","addr":"172.21.15.151:6813","nonce":22285}]},"public_addr":"172.21.15.151:6809/22285","cluster_addr":"172.21.15.151:6811/22285","heartbeat_back_addr":"172.21.15.151:6815/22285","heartbeat_front_addr":"172.21.15.151:6813/22285","state":["exists","up"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":22315},{"type":"v1","addr":"172.21.15.151:6817","nonce":22315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":22315},{"type":"v1","addr":"172.21.15.151:6819","nonce":22315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":22315},{"type":"v1","addr":"172.21.15.151:6823","nonce":22315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":22315},{"type":"v1","addr":"172.21.15.151:6821","nonce":22315}]},"public_addr":"172.21.15.151:6817/22315","cluster_addr":"172.21.15.151:6819/22315","heartbeat_back_addr":"172.21.15.151:6823/22315","heartbeat_front_addr":"172.21.15.151:6821/22315","state":["exists","up"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":81,"up_from":107,"up_thru":137,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":33053},{"type":"v1","addr":"172.21.15.151:6801","nonce":33053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":33053},{"type":"v1","addr":"172.21.15.151:6803","nonce":33053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":33053},{"type":"v1","addr":"172.21.15.151:6807","nonce":33053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":33053},{"type":"v1","addr":"172.21.15.151:6805","nonce":33053}]},"public_addr":"172.21.15.151:6801/33053","cluster_addr":"172.21.15.151:6803/33053","heartbeat_back_addr":"172.21.15.151:6807/33053","heartbeat_front_addr":"172.21.15.151:6805/33053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.191311+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.563551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.493457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.599815+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.634352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.262873+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.508881+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.348276+0000","dead_epoch":82}],"pg_upmap":[{"pgid":"3.1c","osds":[6,1]},{"pgid":"3.1f","osds":[5,3]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:19:39.661 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T13:19:39.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:19:39.948 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:39.949 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":162,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:19:39.331549+0000","last_up_change":"2024-09-16T13:18:43.413345+0000","last_in_change":"2024-09-16T13:17:42.548353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:16:33.497561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:16:40.994538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-16T13:16:50.759705+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":81,"snap_epoch":143,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":22271},{"type":"v1","addr":"172.21.15.98:6809","nonce":22271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":22271},{"type":"v1","addr":"172.21.15.98:6811","nonce":22271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":22271},{"type":"v1","addr":"172.21.15.98:6817","nonce":22271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":22271},{"type":"v1","addr":"172.21.15.98:6814","nonce":22271}]},"public_addr":"172.21.15.98:6809/22271","cluster_addr":"172.21.15.98:6811/22271","heartbeat_back_addr":"172.21.15.98:6817/22271","heartbeat_front_addr":"172.21.15.98:6814/22271","state":["exists","up"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":22272},{"type":"v1","addr":"172.21.15.98:6815","nonce":22272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":22272},{"type":"v1","addr":"172.21.15.98:6819","nonce":22272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":22272},{"type":"v1","addr":"172.21.15.98:6823","nonce":22272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":22272},{"type":"v1","addr":"172.21.15.98:6821","nonce":22272}]},"public_addr":"172.21.15.98:6815/22272","cluster_addr":"172.21.15.98:6819/22272","heartbeat_back_addr":"172.21.15.98:6823/22272","heartbeat_front_addr":"172.21.15.98:6821/22272","state":["exists","up"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":22273},{"type":"v1","addr":"172.21.15.98:6801","nonce":22273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":22273},{"type":"v1","addr":"172.21.15.98:6803","nonce":22273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":22273},{"type":"v1","addr":"172.21.15.98:6807","nonce":22273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":22273},{"type":"v1","addr":"172.21.15.98:6805","nonce":22273}]},"public_addr":"172.21.15.98:6801/22273","cluster_addr":"172.21.15.98:6803/22273","heartbeat_back_addr":"172.21.15.98:6807/22273","heartbeat_front_addr":"172.21.15.98:6805/22273","state":["exists","up"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":22270},{"type":"v1","addr":"172.21.15.98:6825","nonce":22270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":22270},{"type":"v1","addr":"172.21.15.98:6827","nonce":22270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":22270},{"type":"v1","addr":"172.21.15.98:6831","nonce":22270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":22270},{"type":"v1","addr":"172.21.15.98:6829","nonce":22270}]},"public_addr":"172.21.15.98:6825/22270","cluster_addr":"172.21.15.98:6827/22270","heartbeat_back_addr":"172.21.15.98:6831/22270","heartbeat_front_addr":"172.21.15.98:6829/22270","state":["exists","up"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":22286},{"type":"v1","addr":"172.21.15.151:6825","nonce":22286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":22286},{"type":"v1","addr":"172.21.15.151:6827","nonce":22286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":22286},{"type":"v1","addr":"172.21.15.151:6831","nonce":22286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":22286},{"type":"v1","addr":"172.21.15.151:6829","nonce":22286}]},"public_addr":"172.21.15.151:6825/22286","cluster_addr":"172.21.15.151:6827/22286","heartbeat_back_addr":"172.21.15.151:6831/22286","heartbeat_front_addr":"172.21.15.151:6829/22286","state":["exists","up"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":22285},{"type":"v1","addr":"172.21.15.151:6809","nonce":22285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":22285},{"type":"v1","addr":"172.21.15.151:6811","nonce":22285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":22285},{"type":"v1","addr":"172.21.15.151:6815","nonce":22285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":22285},{"type":"v1","addr":"172.21.15.151:6813","nonce":22285}]},"public_addr":"172.21.15.151:6809/22285","cluster_addr":"172.21.15.151:6811/22285","heartbeat_back_addr":"172.21.15.151:6815/22285","heartbeat_front_addr":"172.21.15.151:6813/22285","state":["exists","up"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":22315},{"type":"v1","addr":"172.21.15.151:6817","nonce":22315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":22315},{"type":"v1","addr":"172.21.15.151:6819","nonce":22315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":22315},{"type":"v1","addr":"172.21.15.151:6823","nonce":22315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":22315},{"type":"v1","addr":"172.21.15.151:6821","nonce":22315}]},"public_addr":"172.21.15.151:6817/22315","cluster_addr":"172.21.15.151:6819/22315","heartbeat_back_addr":"172.21.15.151:6823/22315","heartbeat_front_addr":"172.21.15.151:6821/22315","state":["exists","up"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":81,"up_from":107,"up_thru":137,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":33053},{"type":"v1","addr":"172.21.15.151:6801","nonce":33053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":33053},{"type":"v1","addr":"172.21.15.151:6803","nonce":33053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":33053},{"type":"v1","addr":"172.21.15.151:6807","nonce":33053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":33053},{"type":"v1","addr":"172.21.15.151:6805","nonce":33053}]},"public_addr":"172.21.15.151:6801/33053","cluster_addr":"172.21.15.151:6803/33053","heartbeat_back_addr":"172.21.15.151:6807/33053","heartbeat_front_addr":"172.21.15.151:6805/33053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.191311+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.563551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.493457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.599815+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.634352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.262873+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.508881+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.348276+0000","dead_epoch":82}],"pg_upmap":[{"pgid":"3.1c","osds":[6,1]},{"pgid":"3.1f","osds":[5,3]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:19:39.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T13:19:39.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T13:19:39.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T13:19:39.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T13:19:39.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T13:19:39.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T13:19:39.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T13:19:39.966 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T13:19:40.164 INFO:teuthology.orchestra.run.smithi098.stdout:60129542185 2024-09-16T13:19:40.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T13:19:40.165 INFO:teuthology.orchestra.run.smithi098.stdout:60129542185 2024-09-16T13:19:40.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T13:19:40.167 INFO:teuthology.orchestra.run.smithi098.stdout:64424509482 2024-09-16T13:19:40.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T13:19:40.181 INFO:teuthology.orchestra.run.smithi098.stdout:459561500685 2024-09-16T13:19:40.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T13:19:40.182 INFO:teuthology.orchestra.run.smithi098.stdout:64424509482 2024-09-16T13:19:40.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T13:19:40.183 INFO:teuthology.orchestra.run.smithi098.stdout:60129542184 2024-09-16T13:19:40.184 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T13:19:40.185 INFO:teuthology.orchestra.run.smithi098.stdout:64424509482 2024-09-16T13:19:40.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T13:19:40.197 INFO:teuthology.orchestra.run.smithi098.stdout:64424509481 2024-09-16T13:19:40.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T13:19:40.626 INFO:teuthology.orchestra.run.smithi098.stdout:64424509480 2024-09-16T13:19:40.638 INFO:teuthology.orchestra.run.smithi098.stdout:60129542184 2024-09-16T13:19:40.641 INFO:teuthology.orchestra.run.smithi098.stdout:60129542184 2024-09-16T13:19:40.642 INFO:teuthology.orchestra.run.smithi098.stdout:459561500684 2024-09-16T13:19:40.642 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509482 got 64424509480 for osd.3 2024-09-16T13:19:40.655 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542184 for osd.0 2024-09-16T13:19:40.656 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542184 for osd.5 2024-09-16T13:19:40.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 459561500685 got 459561500684 for osd.7 2024-09-16T13:19:40.664 INFO:teuthology.orchestra.run.smithi098.stdout:60129542183 2024-09-16T13:19:40.680 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542184 got 60129542183 for osd.2 2024-09-16T13:19:40.680 INFO:teuthology.orchestra.run.smithi098.stdout:64424509481 2024-09-16T13:19:40.686 INFO:teuthology.orchestra.run.smithi098.stdout:64424509481 2024-09-16T13:19:40.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509482 got 64424509481 for osd.6 2024-09-16T13:19:40.692 INFO:teuthology.orchestra.run.smithi098.stdout:64424509480 2024-09-16T13:19:40.698 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509482 got 64424509481 for osd.1 2024-09-16T13:19:40.704 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509481 got 64424509480 for osd.4 2024-09-16T13:19:41.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T13:19:41.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T13:19:41.656 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T13:19:41.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T13:19:41.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T13:19:41.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T13:19:41.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T13:19:41.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T13:19:42.099 INFO:teuthology.orchestra.run.smithi098.stdout:64424509480 2024-09-16T13:19:42.111 INFO:teuthology.orchestra.run.smithi098.stdout:60129542184 2024-09-16T13:19:42.121 INFO:teuthology.orchestra.run.smithi098.stdout:60129542184 2024-09-16T13:19:42.121 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509482 got 64424509480 for osd.3 2024-09-16T13:19:42.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542184 for osd.0 2024-09-16T13:19:42.135 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542184 for osd.5 2024-09-16T13:19:42.140 INFO:teuthology.orchestra.run.smithi098.stdout:64424509481 2024-09-16T13:19:42.154 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509482 got 64424509481 for osd.6 2024-09-16T13:19:42.164 INFO:teuthology.orchestra.run.smithi098.stdout:60129542183 2024-09-16T13:19:42.165 INFO:teuthology.orchestra.run.smithi098.stdout:459561500684 2024-09-16T13:19:42.177 INFO:tasks.ceph.ceph_manager.ceph:need seq 459561500685 got 459561500684 for osd.7 2024-09-16T13:19:42.180 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542184 got 60129542183 for osd.2 2024-09-16T13:19:42.202 INFO:teuthology.orchestra.run.smithi098.stdout:64424509481 2024-09-16T13:19:42.208 INFO:teuthology.orchestra.run.smithi098.stdout:64424509480 2024-09-16T13:19:42.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509482 got 64424509481 for osd.1 2024-09-16T13:19:42.222 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509481 got 64424509480 for osd.4 2024-09-16T13:19:43.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T13:19:43.131 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T13:19:43.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T13:19:43.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T13:19:43.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T13:19:43.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T13:19:43.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T13:19:43.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T13:19:43.578 INFO:teuthology.orchestra.run.smithi098.stdout:64424509482 2024-09-16T13:19:43.582 INFO:teuthology.orchestra.run.smithi098.stdout:60129542186 2024-09-16T13:19:43.598 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509482 got 64424509482 for osd.3 2024-09-16T13:19:43.598 DEBUG:teuthology.parallel:result is None 2024-09-16T13:19:43.600 INFO:teuthology.orchestra.run.smithi098.stdout:60129542186 2024-09-16T13:19:43.607 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542186 for osd.5 2024-09-16T13:19:43.607 DEBUG:teuthology.parallel:result is None 2024-09-16T13:19:43.616 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542185 got 60129542186 for osd.0 2024-09-16T13:19:43.617 DEBUG:teuthology.parallel:result is None 2024-09-16T13:19:43.626 INFO:teuthology.orchestra.run.smithi098.stdout:459561500686 2024-09-16T13:19:43.640 INFO:tasks.ceph.ceph_manager.ceph:need seq 459561500685 got 459561500686 for osd.7 2024-09-16T13:19:43.640 DEBUG:teuthology.parallel:result is None 2024-09-16T13:19:43.645 INFO:teuthology.orchestra.run.smithi098.stdout:64424509482 2024-09-16T13:19:43.656 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509482 got 64424509482 for osd.6 2024-09-16T13:19:43.656 DEBUG:teuthology.parallel:result is None 2024-09-16T13:19:43.658 INFO:teuthology.orchestra.run.smithi098.stdout:60129542185 2024-09-16T13:19:43.669 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542184 got 60129542185 for osd.2 2024-09-16T13:19:43.670 DEBUG:teuthology.parallel:result is None 2024-09-16T13:19:43.685 INFO:teuthology.orchestra.run.smithi098.stdout:64424509481 2024-09-16T13:19:43.693 INFO:teuthology.orchestra.run.smithi098.stdout:64424509482 2024-09-16T13:19:43.703 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509481 got 64424509481 for osd.4 2024-09-16T13:19:43.703 DEBUG:teuthology.parallel:result is None 2024-09-16T13:19:43.709 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509482 got 64424509482 for osd.1 2024-09-16T13:19:43.709 DEBUG:teuthology.parallel:result is None 2024-09-16T13:19:43.709 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T13:19:43.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:19:43.984 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:19:43.999 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-16T13:19:43.492206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":214,"num_read":143467,"num_read_kb":680580,"num_write":19270,"num_write_kb":297325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":45250401,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16546,"ondisk_log_size":16546,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2454464,"kb_used_data":15284,"kb_used_omap":62,"kb_used_meta":2438977,"kb_avail":752520256,"statfs":{"total":773094113280,"available":770580742144,"internally_reserved":0,"allocated":15650816,"data_stored":8591057,"data_compressed":67468,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":63650,"internal_metadata":2497513310},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.157128"},"pg_stats":[{"pgid":"3.1f","version":"143'370","reported_seq":1980,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473500+0000","last_change":"2024-09-16T13:19:19.135941+0000","last_active":"2024-09-16T13:19:38.473500+0000","last_peered":"2024-09-16T13:19:38.473500+0000","last_clean":"2024-09-16T13:19:38.473500+0000","last_became_active":"2024-09-16T13:18:03.119735+0000","last_became_peered":"2024-09-16T13:18:03.119735+0000","last_unstale":"2024-09-16T13:19:38.473500+0000","last_undegraded":"2024-09-16T13:19:38.473500+0000","last_fullsized":"2024-09-16T13:19:38.473500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:02.061478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012744049999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2248,"num_read_kb":16063,"num_write":361,"num_write_kb":5586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":922127,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"141'824","reported_seq":2700,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336528+0000","last_change":"2024-09-16T13:19:19.131668+0000","last_active":"2024-09-16T13:19:38.336528+0000","last_peered":"2024-09-16T13:19:38.336528+0000","last_clean":"2024-09-16T13:19:38.336528+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:19:38.336528+0000","last_undegraded":"2024-09-16T13:19:38.336528+0000","last_fullsized":"2024-09-16T13:19:38.336528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:50.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003379118,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":5839,"num_read_kb":7327,"num_write":1002,"num_write_kb":6247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"141'446","reported_seq":2670,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473459+0000","last_change":"2024-09-16T13:19:19.136172+0000","last_active":"2024-09-16T13:19:38.473459+0000","last_peered":"2024-09-16T13:19:38.473459+0000","last_clean":"2024-09-16T13:19:38.473459+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:19:38.473459+0000","last_undegraded":"2024-09-16T13:19:38.473459+0000","last_fullsized":"2024-09-16T13:19:38.473459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:14.387827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014771560000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4778,"num_read_kb":13874,"num_write":452,"num_write_kb":6042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"143'449","reported_seq":2727,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.344857+0000","last_change":"2024-09-16T13:19:38.344285+0000","last_active":"2024-09-16T13:19:38.344857+0000","last_peered":"2024-09-16T13:19:38.344857+0000","last_clean":"2024-09-16T13:19:38.344857+0000","last_became_active":"2024-09-16T13:19:38.342560+0000","last_became_peered":"2024-09-16T13:19:38.342560+0000","last_unstale":"2024-09-16T13:19:38.344857+0000","last_undegraded":"2024-09-16T13:19:38.344857+0000","last_fullsized":"2024-09-16T13:19:38.344857+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:57:32.494600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00301937,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":2995,"num_read_kb":25210,"num_write":556,"num_write_kb":12813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"141'671","reported_seq":5617,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343768+0000","last_change":"2024-09-16T13:19:19.164598+0000","last_active":"2024-09-16T13:19:38.343768+0000","last_peered":"2024-09-16T13:19:38.343768+0000","last_clean":"2024-09-16T13:19:38.343768+0000","last_became_active":"2024-09-16T13:19:10.071308+0000","last_became_peered":"2024-09-16T13:19:10.071308+0000","last_unstale":"2024-09-16T13:19:38.343768+0000","last_undegraded":"2024-09-16T13:19:38.343768+0000","last_fullsized":"2024-09-16T13:19:38.343768+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:10:31.562928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002223587,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6572,"num_read_kb":28113,"num_write":920,"num_write_kb":17343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":3647407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"142'661","reported_seq":7962,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338599+0000","last_change":"2024-09-16T13:19:19.129707+0000","last_active":"2024-09-16T13:19:38.338599+0000","last_peered":"2024-09-16T13:19:38.338599+0000","last_clean":"2024-09-16T13:19:38.338599+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:19:38.338599+0000","last_undegraded":"2024-09-16T13:19:38.338599+0000","last_fullsized":"2024-09-16T13:19:38.338599+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:31.071219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028974579999999999,"stat_sum":{"num_bytes":19,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":9256,"num_read_kb":55053,"num_write":857,"num_write_kb":18796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"142'608","reported_seq":5503,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.335998+0000","last_change":"2024-09-16T13:19:19.128105+0000","last_active":"2024-09-16T13:19:38.335998+0000","last_peered":"2024-09-16T13:19:38.335998+0000","last_clean":"2024-09-16T13:19:38.335998+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:19:38.335998+0000","last_undegraded":"2024-09-16T13:19:38.335998+0000","last_fullsized":"2024-09-16T13:19:38.335998+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:42:30.327300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002453879,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":7637,"num_read_kb":32040,"num_write":812,"num_write_kb":13589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'421","reported_seq":2587,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336098+0000","last_change":"2024-09-16T13:19:19.131005+0000","last_active":"2024-09-16T13:19:38.336098+0000","last_peered":"2024-09-16T13:19:38.336098+0000","last_clean":"2024-09-16T13:19:38.336098+0000","last_became_active":"2024-09-16T13:19:10.071112+0000","last_became_peered":"2024-09-16T13:19:10.071112+0000","last_unstale":"2024-09-16T13:19:38.336098+0000","last_undegraded":"2024-09-16T13:19:38.336098+0000","last_fullsized":"2024-09-16T13:19:38.336098+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:29.996047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032828369999999998,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2803,"num_read_kb":8703,"num_write":538,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":228967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"141'573","reported_seq":2387,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343524+0000","last_change":"2024-09-16T13:19:19.163188+0000","last_active":"2024-09-16T13:19:38.343524+0000","last_peered":"2024-09-16T13:19:38.343524+0000","last_clean":"2024-09-16T13:19:38.343524+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:19:38.343524+0000","last_undegraded":"2024-09-16T13:19:38.343524+0000","last_fullsized":"2024-09-16T13:19:38.343524+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:08:57.225261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000869449,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3700,"num_read_kb":13824,"num_write":600,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'410","reported_seq":3104,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473297+0000","last_change":"2024-09-16T13:19:19.135414+0000","last_active":"2024-09-16T13:19:38.473297+0000","last_peered":"2024-09-16T13:19:38.473297+0000","last_clean":"2024-09-16T13:19:38.473297+0000","last_became_active":"2024-09-16T13:18:44.441740+0000","last_became_peered":"2024-09-16T13:18:44.441740+0000","last_unstale":"2024-09-16T13:19:38.473297+0000","last_undegraded":"2024-09-16T13:19:38.473297+0000","last_fullsized":"2024-09-16T13:19:38.473297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:20.994894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082856900000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":5473,"num_read_kb":21309,"num_write":357,"num_write_kb":5655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1185011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"143'229","reported_seq":831,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773785+0000","last_change":"2024-09-16T13:19:19.174836+0000","last_active":"2024-09-16T13:19:38.773785+0000","last_peered":"2024-09-16T13:19:38.773785+0000","last_clean":"2024-09-16T13:19:38.773785+0000","last_became_active":"2024-09-16T13:18:44.435407+0000","last_became_peered":"2024-09-16T13:18:44.435407+0000","last_unstale":"2024-09-16T13:19:38.773785+0000","last_undegraded":"2024-09-16T13:19:38.773785+0000","last_fullsized":"2024-09-16T13:19:38.773785+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:11.546854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016773129999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3249,"num_read_kb":15295,"num_write":190,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1086312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"142'304","reported_seq":5066,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338671+0000","last_change":"2024-09-16T13:19:19.128958+0000","last_active":"2024-09-16T13:19:38.338671+0000","last_peered":"2024-09-16T13:19:38.338671+0000","last_clean":"2024-09-16T13:19:38.338671+0000","last_became_active":"2024-09-16T13:18:47.462532+0000","last_became_peered":"2024-09-16T13:18:47.462532+0000","last_unstale":"2024-09-16T13:19:38.338671+0000","last_undegraded":"2024-09-16T13:19:38.338671+0000","last_fullsized":"2024-09-16T13:19:38.338671+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:16.599784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002111417,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4736,"num_read_kb":25385,"num_write":347,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1053570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.777150+0000","last_change":"2024-09-16T13:19:38.776183+0000","last_active":"2024-09-16T13:19:38.777150+0000","last_peered":"2024-09-16T13:19:38.777150+0000","last_clean":"2024-09-16T13:19:38.777150+0000","last_became_active":"2024-09-16T13:19:38.775280+0000","last_became_peered":"2024-09-16T13:19:38.775280+0000","last_unstale":"2024-09-16T13:19:38.777150+0000","last_undegraded":"2024-09-16T13:19:38.777150+0000","last_fullsized":"2024-09-16T13:19:38.777150+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:46.699452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"143'692","reported_seq":5654,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343415+0000","last_change":"2024-09-16T13:19:19.164842+0000","last_active":"2024-09-16T13:19:38.343415+0000","last_peered":"2024-09-16T13:19:38.343415+0000","last_clean":"2024-09-16T13:19:38.343415+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:19:38.343415+0000","last_undegraded":"2024-09-16T13:19:38.343415+0000","last_fullsized":"2024-09-16T13:19:38.343415+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:52:05.144800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022584039999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6654,"num_read_kb":24625,"num_write":939,"num_write_kb":11031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336150+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:19:38.336150+0000","last_peered":"2024-09-16T13:19:38.336150+0000","last_clean":"2024-09-16T13:19:38.336150+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:19:38.336150+0000","last_undegraded":"2024-09-16T13:19:38.336150+0000","last_fullsized":"2024-09-16T13:19:38.336150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"142'303","reported_seq":1472,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336544+0000","last_change":"2024-09-16T13:19:19.132140+0000","last_active":"2024-09-16T13:19:38.336544+0000","last_peered":"2024-09-16T13:19:38.336544+0000","last_clean":"2024-09-16T13:19:38.336544+0000","last_became_active":"2024-09-16T13:18:47.463039+0000","last_became_peered":"2024-09-16T13:18:47.463039+0000","last_unstale":"2024-09-16T13:19:38.336544+0000","last_undegraded":"2024-09-16T13:19:38.336544+0000","last_fullsized":"2024-09-16T13:19:38.336544+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:05:53.068034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026327310000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":1655,"num_read_kb":8730,"num_write":270,"num_write_kb":5747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1061466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":136,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773757+0000","last_change":"2024-09-16T13:18:44.435471+0000","last_active":"2024-09-16T13:19:38.773757+0000","last_peered":"2024-09-16T13:19:38.773757+0000","last_clean":"2024-09-16T13:19:38.773757+0000","last_became_active":"2024-09-16T13:18:44.434851+0000","last_became_peered":"2024-09-16T13:18:44.434851+0000","last_unstale":"2024-09-16T13:19:38.773757+0000","last_undegraded":"2024-09-16T13:19:38.773757+0000","last_fullsized":"2024-09-16T13:19:38.773757+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:12.980758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"141'372","reported_seq":4687,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336707+0000","last_change":"2024-09-16T13:19:19.129830+0000","last_active":"2024-09-16T13:19:38.336707+0000","last_peered":"2024-09-16T13:19:38.336707+0000","last_clean":"2024-09-16T13:19:38.336707+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:19:38.336707+0000","last_undegraded":"2024-09-16T13:19:38.336707+0000","last_fullsized":"2024-09-16T13:19:38.336707+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:30.996264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032315740000000001,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4693,"num_read_kb":26876,"num_write":422,"num_write_kb":8111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":172,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336904+0000","last_change":"2024-09-16T13:17:42.960492+0000","last_active":"2024-09-16T13:19:38.336904+0000","last_peered":"2024-09-16T13:19:38.336904+0000","last_clean":"2024-09-16T13:19:38.336904+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:19:38.336904+0000","last_undegraded":"2024-09-16T13:19:38.336904+0000","last_fullsized":"2024-09-16T13:19:38.336904+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:35:45.152557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"143'544","reported_seq":3842,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473442+0000","last_change":"2024-09-16T13:19:19.136757+0000","last_active":"2024-09-16T13:19:38.473442+0000","last_peered":"2024-09-16T13:19:38.473442+0000","last_clean":"2024-09-16T13:19:38.473442+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:19:38.473442+0000","last_undegraded":"2024-09-16T13:19:38.473442+0000","last_fullsized":"2024-09-16T13:19:38.473442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:34.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789768,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5947,"num_read_kb":14133,"num_write":543,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"143'506","reported_seq":2497,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.339019+0000","last_change":"2024-09-16T13:19:19.130409+0000","last_active":"2024-09-16T13:19:38.339019+0000","last_peered":"2024-09-16T13:19:38.339019+0000","last_clean":"2024-09-16T13:19:38.339019+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:19:38.339019+0000","last_undegraded":"2024-09-16T13:19:38.339019+0000","last_fullsized":"2024-09-16T13:19:38.339019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:44:16.402879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003357956,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2667,"num_read_kb":21275,"num_write":487,"num_write_kb":9028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":156,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473170+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:19:38.473170+0000","last_peered":"2024-09-16T13:19:38.473170+0000","last_clean":"2024-09-16T13:19:38.473170+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:19:38.473170+0000","last_undegraded":"2024-09-16T13:19:38.473170+0000","last_fullsized":"2024-09-16T13:19:38.473170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"142'424","reported_seq":1787,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.335712+0000","last_change":"2024-09-16T13:19:19.127890+0000","last_active":"2024-09-16T13:19:38.335712+0000","last_peered":"2024-09-16T13:19:38.335712+0000","last_clean":"2024-09-16T13:19:38.335712+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:19:38.335712+0000","last_undegraded":"2024-09-16T13:19:38.335712+0000","last_fullsized":"2024-09-16T13:19:38.335712+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:29.090161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020916900000000002,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3126,"num_read_kb":11030,"num_write":475,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773196+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:19:38.773196+0000","last_peered":"2024-09-16T13:19:38.773196+0000","last_clean":"2024-09-16T13:19:38.773196+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:19:38.773196+0000","last_undegraded":"2024-09-16T13:19:38.773196+0000","last_fullsized":"2024-09-16T13:19:38.773196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'709","reported_seq":2213,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343225+0000","last_change":"2024-09-16T13:19:19.164008+0000","last_active":"2024-09-16T13:19:38.343225+0000","last_peered":"2024-09-16T13:19:38.343225+0000","last_clean":"2024-09-16T13:19:38.343225+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:19:38.343225+0000","last_undegraded":"2024-09-16T13:19:38.343225+0000","last_fullsized":"2024-09-16T13:19:38.343225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:03.357310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017855869999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3881,"num_read_kb":31185,"num_write":963,"num_write_kb":16374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773578+0000","last_change":"2024-09-16T13:18:44.435672+0000","last_active":"2024-09-16T13:19:38.773578+0000","last_peered":"2024-09-16T13:19:38.773578+0000","last_clean":"2024-09-16T13:19:38.773578+0000","last_became_active":"2024-09-16T13:18:44.435188+0000","last_became_peered":"2024-09-16T13:18:44.435188+0000","last_unstale":"2024-09-16T13:19:38.773578+0000","last_undegraded":"2024-09-16T13:19:38.773578+0000","last_fullsized":"2024-09-16T13:19:38.773578+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:35.400164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"143'487","reported_seq":2228,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336304+0000","last_change":"2024-09-16T13:19:19.131189+0000","last_active":"2024-09-16T13:19:38.336304+0000","last_peered":"2024-09-16T13:19:38.336304+0000","last_clean":"2024-09-16T13:19:38.336304+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:19:38.336304+0000","last_undegraded":"2024-09-16T13:19:38.336304+0000","last_fullsized":"2024-09-16T13:19:38.336304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:45.154984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003557705,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3512,"num_read_kb":15515,"num_write":504,"num_write_kb":6261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473138+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:19:38.473138+0000","last_peered":"2024-09-16T13:19:38.473138+0000","last_clean":"2024-09-16T13:19:38.473138+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:19:38.473138+0000","last_undegraded":"2024-09-16T13:19:38.473138+0000","last_fullsized":"2024-09-16T13:19:38.473138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":236,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773593+0000","last_change":"2024-09-16T13:18:44.437802+0000","last_active":"2024-09-16T13:19:38.773593+0000","last_peered":"2024-09-16T13:19:38.773593+0000","last_clean":"2024-09-16T13:19:38.773593+0000","last_became_active":"2024-09-16T13:18:44.437418+0000","last_became_peered":"2024-09-16T13:18:44.437418+0000","last_unstale":"2024-09-16T13:19:38.773593+0000","last_undegraded":"2024-09-16T13:19:38.773593+0000","last_fullsized":"2024-09-16T13:19:38.773593+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:29.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"143'606","reported_seq":1309,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336209+0000","last_change":"2024-09-16T13:19:19.131515+0000","last_active":"2024-09-16T13:19:38.336209+0000","last_peered":"2024-09-16T13:19:38.336209+0000","last_clean":"2024-09-16T13:19:38.336209+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:19:38.336209+0000","last_undegraded":"2024-09-16T13:19:38.336209+0000","last_fullsized":"2024-09-16T13:19:38.336209+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:59.515115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033814320000000002,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2017,"num_read_kb":19372,"num_write":789,"num_write_kb":12171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"142'523","reported_seq":3009,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338761+0000","last_change":"2024-09-16T13:19:19.128362+0000","last_active":"2024-09-16T13:19:38.338761+0000","last_peered":"2024-09-16T13:19:38.338761+0000","last_clean":"2024-09-16T13:19:38.338761+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:19:38.338761+0000","last_undegraded":"2024-09-16T13:19:38.338761+0000","last_fullsized":"2024-09-16T13:19:38.338761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:49:10.301550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001533006,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4183,"num_read_kb":31351,"num_write":445,"num_write_kb":12351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"142'942","reported_seq":3022,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.347467+0000","last_change":"2024-09-16T13:19:38.346887+0000","last_active":"2024-09-16T13:19:38.347467+0000","last_peered":"2024-09-16T13:19:38.347467+0000","last_clean":"2024-09-16T13:19:38.347467+0000","last_became_active":"2024-09-16T13:19:38.343200+0000","last_became_peered":"2024-09-16T13:19:38.343200+0000","last_unstale":"2024-09-16T13:19:38.347467+0000","last_undegraded":"2024-09-16T13:19:38.347467+0000","last_fullsized":"2024-09-16T13:19:38.347467+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:37.001790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025018559999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":4754,"num_read_kb":19969,"num_write":1275,"num_write_kb":12323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"142'1140","reported_seq":4459,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336696+0000","last_change":"2024-09-16T13:19:19.128947+0000","last_active":"2024-09-16T13:19:38.336696+0000","last_peered":"2024-09-16T13:19:38.336696+0000","last_clean":"2024-09-16T13:19:38.336696+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:19:38.336696+0000","last_undegraded":"2024-09-16T13:19:38.336696+0000","last_fullsized":"2024-09-16T13:19:38.336696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:20.684083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014023239999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":9870,"num_read_kb":27372,"num_write":1699,"num_write_kb":10648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"143'323","reported_seq":1457,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473160+0000","last_change":"2024-09-16T13:19:19.135617+0000","last_active":"2024-09-16T13:19:38.473160+0000","last_peered":"2024-09-16T13:19:38.473160+0000","last_clean":"2024-09-16T13:19:38.473160+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:19:38.473160+0000","last_undegraded":"2024-09-16T13:19:38.473160+0000","last_fullsized":"2024-09-16T13:19:38.473160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:39.937083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010610159999999999,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":1447,"num_read_kb":15039,"num_write":293,"num_write_kb":7034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'389","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774008+0000","last_change":"2024-09-16T13:19:19.175390+0000","last_active":"2024-09-16T13:19:38.774008+0000","last_peered":"2024-09-16T13:19:38.774008+0000","last_clean":"2024-09-16T13:19:38.774008+0000","last_became_active":"2024-09-16T13:18:47.468001+0000","last_became_peered":"2024-09-16T13:18:47.468001+0000","last_unstale":"2024-09-16T13:19:38.774008+0000","last_undegraded":"2024-09-16T13:19:38.774008+0000","last_fullsized":"2024-09-16T13:19:38.774008+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:11:32.886561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018957069999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2747,"num_read_kb":18459,"num_write":343,"num_write_kb":14196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"142'510","reported_seq":7096,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773857+0000","last_change":"2024-09-16T13:19:19.173823+0000","last_active":"2024-09-16T13:19:38.773857+0000","last_peered":"2024-09-16T13:19:38.773857+0000","last_clean":"2024-09-16T13:19:38.773857+0000","last_became_active":"2024-09-16T13:18:47.467028+0000","last_became_peered":"2024-09-16T13:18:47.467028+0000","last_unstale":"2024-09-16T13:19:38.773857+0000","last_undegraded":"2024-09-16T13:19:38.773857+0000","last_fullsized":"2024-09-16T13:19:38.773857+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:35.852378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092761899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":8104,"num_read_kb":29095,"num_write":570,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1715710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"142'310","reported_seq":1005,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774295+0000","last_change":"2024-09-16T13:19:19.173006+0000","last_active":"2024-09-16T13:19:38.774295+0000","last_peered":"2024-09-16T13:19:38.774295+0000","last_clean":"2024-09-16T13:19:38.774295+0000","last_became_active":"2024-09-16T13:18:44.444025+0000","last_became_peered":"2024-09-16T13:18:44.444025+0000","last_unstale":"2024-09-16T13:19:38.774295+0000","last_undegraded":"2024-09-16T13:19:38.774295+0000","last_fullsized":"2024-09-16T13:19:38.774295+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:25:03.521529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056513400000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":692,"num_read_kb":10388,"num_write":284,"num_write_kb":7715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":63719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"142'265","reported_seq":1394,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.346281+0000","last_change":"2024-09-16T13:19:38.345793+0000","last_active":"2024-09-16T13:19:38.346281+0000","last_peered":"2024-09-16T13:19:38.346281+0000","last_clean":"2024-09-16T13:19:38.346281+0000","last_became_active":"2024-09-16T13:19:38.344229+0000","last_became_peered":"2024-09-16T13:19:38.344229+0000","last_unstale":"2024-09-16T13:19:38.346281+0000","last_undegraded":"2024-09-16T13:19:38.346281+0000","last_fullsized":"2024-09-16T13:19:38.346281+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:30:56.026666+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00077869099999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":784,"num_read_kb":16305,"num_write":257,"num_write_kb":11082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"142'556","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773502+0000","last_change":"2024-09-16T13:19:19.175226+0000","last_active":"2024-09-16T13:19:38.773502+0000","last_peered":"2024-09-16T13:19:38.773502+0000","last_clean":"2024-09-16T13:19:38.773502+0000","last_became_active":"2024-09-16T13:18:47.462561+0000","last_became_peered":"2024-09-16T13:18:47.462561+0000","last_unstale":"2024-09-16T13:19:38.773502+0000","last_undegraded":"2024-09-16T13:19:38.773502+0000","last_fullsized":"2024-09-16T13:19:38.773502+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.079652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021368229999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3858,"num_read_kb":23843,"num_write":732,"num_write_kb":11756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"141'398","reported_seq":7509,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343225+0000","last_change":"2024-09-16T13:19:19.164016+0000","last_active":"2024-09-16T13:19:38.343225+0000","last_peered":"2024-09-16T13:19:38.343225+0000","last_clean":"2024-09-16T13:19:38.343225+0000","last_became_active":"2024-09-16T13:18:47.462412+0000","last_became_peered":"2024-09-16T13:18:47.462412+0000","last_unstale":"2024-09-16T13:19:38.343225+0000","last_undegraded":"2024-09-16T13:19:38.343225+0000","last_fullsized":"2024-09-16T13:19:38.343225+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:50:25.256657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001704484,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":8501,"num_read_kb":35036,"num_write":362,"num_write_kb":9220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2494648,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"141'506","reported_seq":2090,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773579+0000","last_change":"2024-09-16T13:19:19.172960+0000","last_active":"2024-09-16T13:19:38.773579+0000","last_peered":"2024-09-16T13:19:38.773579+0000","last_clean":"2024-09-16T13:19:38.773579+0000","last_became_active":"2024-09-16T13:18:44.443832+0000","last_became_peered":"2024-09-16T13:18:44.443832+0000","last_unstale":"2024-09-16T13:19:38.773579+0000","last_undegraded":"2024-09-16T13:19:38.773579+0000","last_fullsized":"2024-09-16T13:19:38.773579+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:45.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001469344,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":5023,"num_read_kb":18731,"num_write":523,"num_write_kb":7705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1653183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":218,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":214,"num_read":143401,"num_read_kb":680525,"num_write":19167,"num_write_kb":295695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":44660033,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0},"log_size":16471,"ondisk_log_size":16471,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509482,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311724,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060116,"statfs":{"total":96636764160,"available":96317558784,"internally_reserved":0,"allocated":1929216,"data_stored":921154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.84999999999999998},"min":{"1min":0.47299999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2809999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.499,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1910000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":3.2130000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.83799999999999997,"15min":0.82799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.3009999999999999,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.98799999999999999,"15min":0.98199999999999998},"min":{"1min":0.64500000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.6789999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":1.1060000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.878,"15min":0.89400000000000002},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.268,"5min":1.476,"15min":1.476},"last":1.319},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.97699999999999998,"15min":1.0389999999999999},"min":{"1min":0.63300000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.278,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":3.1299999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.0409999999999999,"15min":1.081},"min":{"1min":0.73099999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.5350000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":1.4119999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.98599999999999999,"15min":0.999},"min":{"1min":0.65200000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.256,"5min":3.0209999999999999,"15min":3.0209999999999999},"last":2.9860000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.96599999999999997,"15min":0.98799999999999999},"min":{"1min":0.53800000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.335,"5min":1.825,"15min":1.825},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0309999999999999,"15min":1.0600000000000001},"min":{"1min":0.56299999999999994,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.347,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.004}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88100000000000001,"15min":0.90300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.409,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.115,"5min":1.1100000000000001,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":1.1559999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.24},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":3.286}]}]},{"osd":4,"up_from":15,"seq":64424509481,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311740,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060100,"statfs":{"total":96636764160,"available":96317542400,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.78300000000000003,"15min":0.80100000000000005},"min":{"1min":0.51600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.927,"15min":1.927},"last":1.282}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78700000000000003,"15min":0.78300000000000003},"min":{"1min":0.52300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.90000000000000002,"15min":0.91600000000000004},"min":{"1min":0.55000000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.3109999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.82499999999999996,"15min":0.82199999999999995},"min":{"1min":0.59699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.93500000000000005},"min":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0209999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.83599999999999997,"15min":0.84399999999999997},"min":{"1min":0.69699999999999995,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.167},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.94699999999999995,"15min":0.95399999999999996},"min":{"1min":0.85999999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.1339999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.96199999999999997},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.4239999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.341,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94599999999999995,"15min":0.97999999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.0609999999999999,"15min":1.0920000000000001},"min":{"1min":0.59899999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.369}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.2490000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509482,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311736,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060104,"statfs":{"total":96636764160,"available":96317546496,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.45400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.59499999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65000000000000002,"15min":0.66300000000000003},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92600000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.76600000000000001},"min":{"1min":0.48399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0660000000000001,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.76300000000000001,"15min":0.77100000000000002},"min":{"1min":0.57399999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.88500000000000001},"min":{"1min":0.56399999999999995,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1879999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.98599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0209999999999999,"15min":0.98899999999999999},"min":{"1min":0.747,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.7989999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":1.177},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.98999999999999999,"15min":0.96499999999999997},"min":{"1min":0.77500000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3100000000000001,"5min":4.9489999999999998,"15min":4.9489999999999998},"last":1.3100000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.99299999999999999,"15min":0.95599999999999996},"min":{"1min":0.71499999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3380000000000001,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":1.147},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.087,"15min":1.091},"min":{"1min":0.86499999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.45,"15min":1.45},"last":1.2849999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0649999999999999,"15min":1.0629999999999999},"min":{"1min":0.69999999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.843,"5min":2.843,"15min":2.843},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.01,"15min":1.01},"min":{"1min":0.80300000000000005,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.385}]},{"osd":7,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.2490000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509482,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303428,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068412,"statfs":{"total":96636764160,"available":96326053888,"internally_reserved":0,"allocated":1949696,"data_stored":927051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.911,"5min":3.911,"15min":3.911},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.46600000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9729999999999999,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":0.96099999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9510000000000001,"5min":2.9510000000000001,"15min":2.9510000000000001},"last":0.88}]},{"osd":3,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.75800000000000001,"15min":0.72799999999999998},"min":{"1min":0.58199999999999996,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.2089999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":0.80500000000000005,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.76500000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.86899999999999999,"15min":0.84399999999999997},"min":{"1min":0.56299999999999994,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.014},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.96799999999999997,"15min":0.96699999999999997},"min":{"1min":0.51500000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.343}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":0.97499999999999998,"15min":0.94099999999999995},"min":{"1min":0.86499999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":5.0919999999999996,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":1.101},{"interface":"front","average":{"1min":1.163,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.61899999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":3.8279999999999998},"last":0.94999999999999996}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.0429999999999999,"15min":1.0089999999999999},"min":{"1min":0.75700000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.04,"15min":1.0269999999999999},"min":{"1min":0.58299999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":4.0460000000000003,"5min":4.0460000000000003,"15min":4.0460000000000003},"last":1.085}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.995},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.288}]}]},{"osd":7,"up_from":107,"seq":459561500686,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305528,"kb_used_data":1960,"kb_used_omap":50,"kb_used_meta":303501,"kb_avail":94066312,"statfs":{"total":96636764160,"available":96323903488,"internally_reserved":0,"allocated":2007040,"data_stored":1517381,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":51722,"internal_metadata":310785526},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299264,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072576,"statfs":{"total":96636764160,"available":96330317824,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.77800000000000002,"15min":0.76600000000000001},"min":{"1min":0.57399999999999995,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2929999999999999,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96999999999999997,"15min":0.94799999999999995},"min":{"1min":0.52300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.335,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":1.016}]},{"osd":1,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.78500000000000003,"15min":0.77300000000000002},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86899999999999999,"15min":0.86499999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.81899999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84599999999999997,"15min":0.86599999999999999},"min":{"1min":0.60799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.1579999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.044,"5min":0.95999999999999996,"15min":0.95899999999999996},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3979999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.88200000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.97599999999999998,"15min":0.95799999999999996},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.107}]},{"osd":4,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.871,"15min":0.88700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1220000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.874,"5min":1.002,"15min":1.0229999999999999},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3009999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.2509999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.96599999999999997,"15min":0.97799999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.329,"5min":1.637,"15min":1.637},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0529999999999999,"15min":1.0720000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.1910000000000001,"5min":1.661,"15min":1.661},"last":1.1950000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.171},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.224}]}]},{"osd":2,"up_from":14,"seq":60129542185,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307580,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064260,"statfs":{"total":96636764160,"available":96321802240,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59699999999999998,"15min":0.60299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85599999999999998,"5min":1.159,"15min":1.159},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.38600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.506,"15min":1.506},"last":0.33400000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.438,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.05,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.218,"5min":3.298,"15min":3.298},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79600000000000004,"15min":0.80700000000000005},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0980000000000001,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.91800000000000004},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.266,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.97999999999999998,"15min":1.014},"min":{"1min":0.61499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.5780000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.05,"5min":1.002,"15min":1},"min":{"1min":0.57299999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.6279999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.92600000000000005}]},{"osd":5,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0569999999999999,"15min":1.081},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.726,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.131,"5min":1.0880000000000001,"15min":1.1060000000000001},"min":{"1min":0.69799999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5029999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.131}]},{"osd":6,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.054,"15min":1.069},"min":{"1min":0.67600000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.758,"5min":1.857,"15min":1.857},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.141,"15min":1.123},"min":{"1min":0.64600000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4079999999999999,"5min":7.0090000000000003,"15min":7.0090000000000003},"last":1.1779999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.056},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":0.63800000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303464,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068376,"statfs":{"total":96636764160,"available":96326017024,"internally_reserved":0,"allocated":1990656,"data_stored":1517381,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.60699999999999998,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.501,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.01,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.70099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.224,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.53400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.109,"5min":1.74,"15min":1.74},"last":0.91100000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.76200000000000001,"15min":0.77600000000000002},"min":{"1min":0.50900000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.48099999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.276,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1}]},{"osd":4,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.92700000000000005,"15min":0.92500000000000004},"min":{"1min":0.73399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.615,"5min":1.871,"15min":1.871},"last":1.048},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.71499999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.2809999999999999,"5min":1.903,"15min":1.903},"last":0.86199999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0269999999999999,"15min":1.0069999999999999},"min":{"1min":0.80100000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.357,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.242},{"interface":"front","average":{"1min":1.073,"5min":0.97999999999999998,"15min":0.96299999999999997},"min":{"1min":0.70099999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.103}]},{"osd":6,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.044,"15min":1.0409999999999999},"min":{"1min":0.71299999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3879999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0369999999999999,"15min":1.0269999999999999},"min":{"1min":0.75600000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5149999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.0740000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.881,"5min":1.881,"15min":1.881},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0}]}} 2024-09-16T13:19:44.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:19:44.281 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:44.283 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:19:44.299 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-16T13:19:43.492206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":214,"num_read":143467,"num_read_kb":680580,"num_write":19270,"num_write_kb":297325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":45250401,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16546,"ondisk_log_size":16546,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2454464,"kb_used_data":15284,"kb_used_omap":62,"kb_used_meta":2438977,"kb_avail":752520256,"statfs":{"total":773094113280,"available":770580742144,"internally_reserved":0,"allocated":15650816,"data_stored":8591057,"data_compressed":67468,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":63650,"internal_metadata":2497513310},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.157128"},"pg_stats":[{"pgid":"3.1f","version":"143'370","reported_seq":1980,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473500+0000","last_change":"2024-09-16T13:19:19.135941+0000","last_active":"2024-09-16T13:19:38.473500+0000","last_peered":"2024-09-16T13:19:38.473500+0000","last_clean":"2024-09-16T13:19:38.473500+0000","last_became_active":"2024-09-16T13:18:03.119735+0000","last_became_peered":"2024-09-16T13:18:03.119735+0000","last_unstale":"2024-09-16T13:19:38.473500+0000","last_undegraded":"2024-09-16T13:19:38.473500+0000","last_fullsized":"2024-09-16T13:19:38.473500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:02.061478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012744049999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2248,"num_read_kb":16063,"num_write":361,"num_write_kb":5586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":922127,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"141'824","reported_seq":2700,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336528+0000","last_change":"2024-09-16T13:19:19.131668+0000","last_active":"2024-09-16T13:19:38.336528+0000","last_peered":"2024-09-16T13:19:38.336528+0000","last_clean":"2024-09-16T13:19:38.336528+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:19:38.336528+0000","last_undegraded":"2024-09-16T13:19:38.336528+0000","last_fullsized":"2024-09-16T13:19:38.336528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:50.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003379118,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":5839,"num_read_kb":7327,"num_write":1002,"num_write_kb":6247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"141'446","reported_seq":2670,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473459+0000","last_change":"2024-09-16T13:19:19.136172+0000","last_active":"2024-09-16T13:19:38.473459+0000","last_peered":"2024-09-16T13:19:38.473459+0000","last_clean":"2024-09-16T13:19:38.473459+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:19:38.473459+0000","last_undegraded":"2024-09-16T13:19:38.473459+0000","last_fullsized":"2024-09-16T13:19:38.473459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:14.387827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014771560000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4778,"num_read_kb":13874,"num_write":452,"num_write_kb":6042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"143'449","reported_seq":2727,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.344857+0000","last_change":"2024-09-16T13:19:38.344285+0000","last_active":"2024-09-16T13:19:38.344857+0000","last_peered":"2024-09-16T13:19:38.344857+0000","last_clean":"2024-09-16T13:19:38.344857+0000","last_became_active":"2024-09-16T13:19:38.342560+0000","last_became_peered":"2024-09-16T13:19:38.342560+0000","last_unstale":"2024-09-16T13:19:38.344857+0000","last_undegraded":"2024-09-16T13:19:38.344857+0000","last_fullsized":"2024-09-16T13:19:38.344857+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:57:32.494600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00301937,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":2995,"num_read_kb":25210,"num_write":556,"num_write_kb":12813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"141'671","reported_seq":5617,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343768+0000","last_change":"2024-09-16T13:19:19.164598+0000","last_active":"2024-09-16T13:19:38.343768+0000","last_peered":"2024-09-16T13:19:38.343768+0000","last_clean":"2024-09-16T13:19:38.343768+0000","last_became_active":"2024-09-16T13:19:10.071308+0000","last_became_peered":"2024-09-16T13:19:10.071308+0000","last_unstale":"2024-09-16T13:19:38.343768+0000","last_undegraded":"2024-09-16T13:19:38.343768+0000","last_fullsized":"2024-09-16T13:19:38.343768+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:10:31.562928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002223587,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6572,"num_read_kb":28113,"num_write":920,"num_write_kb":17343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":3647407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"142'661","reported_seq":7962,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338599+0000","last_change":"2024-09-16T13:19:19.129707+0000","last_active":"2024-09-16T13:19:38.338599+0000","last_peered":"2024-09-16T13:19:38.338599+0000","last_clean":"2024-09-16T13:19:38.338599+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:19:38.338599+0000","last_undegraded":"2024-09-16T13:19:38.338599+0000","last_fullsized":"2024-09-16T13:19:38.338599+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:31.071219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028974579999999999,"stat_sum":{"num_bytes":19,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":9256,"num_read_kb":55053,"num_write":857,"num_write_kb":18796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"142'608","reported_seq":5503,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.335998+0000","last_change":"2024-09-16T13:19:19.128105+0000","last_active":"2024-09-16T13:19:38.335998+0000","last_peered":"2024-09-16T13:19:38.335998+0000","last_clean":"2024-09-16T13:19:38.335998+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:19:38.335998+0000","last_undegraded":"2024-09-16T13:19:38.335998+0000","last_fullsized":"2024-09-16T13:19:38.335998+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:42:30.327300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002453879,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":7637,"num_read_kb":32040,"num_write":812,"num_write_kb":13589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'421","reported_seq":2587,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336098+0000","last_change":"2024-09-16T13:19:19.131005+0000","last_active":"2024-09-16T13:19:38.336098+0000","last_peered":"2024-09-16T13:19:38.336098+0000","last_clean":"2024-09-16T13:19:38.336098+0000","last_became_active":"2024-09-16T13:19:10.071112+0000","last_became_peered":"2024-09-16T13:19:10.071112+0000","last_unstale":"2024-09-16T13:19:38.336098+0000","last_undegraded":"2024-09-16T13:19:38.336098+0000","last_fullsized":"2024-09-16T13:19:38.336098+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:29.996047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032828369999999998,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2803,"num_read_kb":8703,"num_write":538,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":228967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"141'573","reported_seq":2387,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343524+0000","last_change":"2024-09-16T13:19:19.163188+0000","last_active":"2024-09-16T13:19:38.343524+0000","last_peered":"2024-09-16T13:19:38.343524+0000","last_clean":"2024-09-16T13:19:38.343524+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:19:38.343524+0000","last_undegraded":"2024-09-16T13:19:38.343524+0000","last_fullsized":"2024-09-16T13:19:38.343524+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:08:57.225261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000869449,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3700,"num_read_kb":13824,"num_write":600,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'410","reported_seq":3104,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473297+0000","last_change":"2024-09-16T13:19:19.135414+0000","last_active":"2024-09-16T13:19:38.473297+0000","last_peered":"2024-09-16T13:19:38.473297+0000","last_clean":"2024-09-16T13:19:38.473297+0000","last_became_active":"2024-09-16T13:18:44.441740+0000","last_became_peered":"2024-09-16T13:18:44.441740+0000","last_unstale":"2024-09-16T13:19:38.473297+0000","last_undegraded":"2024-09-16T13:19:38.473297+0000","last_fullsized":"2024-09-16T13:19:38.473297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:20.994894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082856900000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":5473,"num_read_kb":21309,"num_write":357,"num_write_kb":5655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1185011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"143'229","reported_seq":831,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773785+0000","last_change":"2024-09-16T13:19:19.174836+0000","last_active":"2024-09-16T13:19:38.773785+0000","last_peered":"2024-09-16T13:19:38.773785+0000","last_clean":"2024-09-16T13:19:38.773785+0000","last_became_active":"2024-09-16T13:18:44.435407+0000","last_became_peered":"2024-09-16T13:18:44.435407+0000","last_unstale":"2024-09-16T13:19:38.773785+0000","last_undegraded":"2024-09-16T13:19:38.773785+0000","last_fullsized":"2024-09-16T13:19:38.773785+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:11.546854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016773129999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3249,"num_read_kb":15295,"num_write":190,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1086312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"142'304","reported_seq":5066,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338671+0000","last_change":"2024-09-16T13:19:19.128958+0000","last_active":"2024-09-16T13:19:38.338671+0000","last_peered":"2024-09-16T13:19:38.338671+0000","last_clean":"2024-09-16T13:19:38.338671+0000","last_became_active":"2024-09-16T13:18:47.462532+0000","last_became_peered":"2024-09-16T13:18:47.462532+0000","last_unstale":"2024-09-16T13:19:38.338671+0000","last_undegraded":"2024-09-16T13:19:38.338671+0000","last_fullsized":"2024-09-16T13:19:38.338671+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:16.599784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002111417,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4736,"num_read_kb":25385,"num_write":347,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1053570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.777150+0000","last_change":"2024-09-16T13:19:38.776183+0000","last_active":"2024-09-16T13:19:38.777150+0000","last_peered":"2024-09-16T13:19:38.777150+0000","last_clean":"2024-09-16T13:19:38.777150+0000","last_became_active":"2024-09-16T13:19:38.775280+0000","last_became_peered":"2024-09-16T13:19:38.775280+0000","last_unstale":"2024-09-16T13:19:38.777150+0000","last_undegraded":"2024-09-16T13:19:38.777150+0000","last_fullsized":"2024-09-16T13:19:38.777150+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:46.699452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"143'692","reported_seq":5654,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343415+0000","last_change":"2024-09-16T13:19:19.164842+0000","last_active":"2024-09-16T13:19:38.343415+0000","last_peered":"2024-09-16T13:19:38.343415+0000","last_clean":"2024-09-16T13:19:38.343415+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:19:38.343415+0000","last_undegraded":"2024-09-16T13:19:38.343415+0000","last_fullsized":"2024-09-16T13:19:38.343415+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:52:05.144800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022584039999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6654,"num_read_kb":24625,"num_write":939,"num_write_kb":11031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336150+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:19:38.336150+0000","last_peered":"2024-09-16T13:19:38.336150+0000","last_clean":"2024-09-16T13:19:38.336150+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:19:38.336150+0000","last_undegraded":"2024-09-16T13:19:38.336150+0000","last_fullsized":"2024-09-16T13:19:38.336150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"142'303","reported_seq":1472,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336544+0000","last_change":"2024-09-16T13:19:19.132140+0000","last_active":"2024-09-16T13:19:38.336544+0000","last_peered":"2024-09-16T13:19:38.336544+0000","last_clean":"2024-09-16T13:19:38.336544+0000","last_became_active":"2024-09-16T13:18:47.463039+0000","last_became_peered":"2024-09-16T13:18:47.463039+0000","last_unstale":"2024-09-16T13:19:38.336544+0000","last_undegraded":"2024-09-16T13:19:38.336544+0000","last_fullsized":"2024-09-16T13:19:38.336544+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:05:53.068034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026327310000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":1655,"num_read_kb":8730,"num_write":270,"num_write_kb":5747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1061466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":136,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773757+0000","last_change":"2024-09-16T13:18:44.435471+0000","last_active":"2024-09-16T13:19:38.773757+0000","last_peered":"2024-09-16T13:19:38.773757+0000","last_clean":"2024-09-16T13:19:38.773757+0000","last_became_active":"2024-09-16T13:18:44.434851+0000","last_became_peered":"2024-09-16T13:18:44.434851+0000","last_unstale":"2024-09-16T13:19:38.773757+0000","last_undegraded":"2024-09-16T13:19:38.773757+0000","last_fullsized":"2024-09-16T13:19:38.773757+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:12.980758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"141'372","reported_seq":4687,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336707+0000","last_change":"2024-09-16T13:19:19.129830+0000","last_active":"2024-09-16T13:19:38.336707+0000","last_peered":"2024-09-16T13:19:38.336707+0000","last_clean":"2024-09-16T13:19:38.336707+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:19:38.336707+0000","last_undegraded":"2024-09-16T13:19:38.336707+0000","last_fullsized":"2024-09-16T13:19:38.336707+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:30.996264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032315740000000001,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4693,"num_read_kb":26876,"num_write":422,"num_write_kb":8111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":172,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336904+0000","last_change":"2024-09-16T13:17:42.960492+0000","last_active":"2024-09-16T13:19:38.336904+0000","last_peered":"2024-09-16T13:19:38.336904+0000","last_clean":"2024-09-16T13:19:38.336904+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:19:38.336904+0000","last_undegraded":"2024-09-16T13:19:38.336904+0000","last_fullsized":"2024-09-16T13:19:38.336904+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:35:45.152557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"143'544","reported_seq":3842,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473442+0000","last_change":"2024-09-16T13:19:19.136757+0000","last_active":"2024-09-16T13:19:38.473442+0000","last_peered":"2024-09-16T13:19:38.473442+0000","last_clean":"2024-09-16T13:19:38.473442+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:19:38.473442+0000","last_undegraded":"2024-09-16T13:19:38.473442+0000","last_fullsized":"2024-09-16T13:19:38.473442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:34.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789768,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5947,"num_read_kb":14133,"num_write":543,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"143'506","reported_seq":2497,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.339019+0000","last_change":"2024-09-16T13:19:19.130409+0000","last_active":"2024-09-16T13:19:38.339019+0000","last_peered":"2024-09-16T13:19:38.339019+0000","last_clean":"2024-09-16T13:19:38.339019+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:19:38.339019+0000","last_undegraded":"2024-09-16T13:19:38.339019+0000","last_fullsized":"2024-09-16T13:19:38.339019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:44:16.402879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003357956,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2667,"num_read_kb":21275,"num_write":487,"num_write_kb":9028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":156,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473170+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:19:38.473170+0000","last_peered":"2024-09-16T13:19:38.473170+0000","last_clean":"2024-09-16T13:19:38.473170+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:19:38.473170+0000","last_undegraded":"2024-09-16T13:19:38.473170+0000","last_fullsized":"2024-09-16T13:19:38.473170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"142'424","reported_seq":1787,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.335712+0000","last_change":"2024-09-16T13:19:19.127890+0000","last_active":"2024-09-16T13:19:38.335712+0000","last_peered":"2024-09-16T13:19:38.335712+0000","last_clean":"2024-09-16T13:19:38.335712+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:19:38.335712+0000","last_undegraded":"2024-09-16T13:19:38.335712+0000","last_fullsized":"2024-09-16T13:19:38.335712+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:29.090161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020916900000000002,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3126,"num_read_kb":11030,"num_write":475,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773196+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:19:38.773196+0000","last_peered":"2024-09-16T13:19:38.773196+0000","last_clean":"2024-09-16T13:19:38.773196+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:19:38.773196+0000","last_undegraded":"2024-09-16T13:19:38.773196+0000","last_fullsized":"2024-09-16T13:19:38.773196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'709","reported_seq":2213,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343225+0000","last_change":"2024-09-16T13:19:19.164008+0000","last_active":"2024-09-16T13:19:38.343225+0000","last_peered":"2024-09-16T13:19:38.343225+0000","last_clean":"2024-09-16T13:19:38.343225+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:19:38.343225+0000","last_undegraded":"2024-09-16T13:19:38.343225+0000","last_fullsized":"2024-09-16T13:19:38.343225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:03.357310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017855869999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3881,"num_read_kb":31185,"num_write":963,"num_write_kb":16374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773578+0000","last_change":"2024-09-16T13:18:44.435672+0000","last_active":"2024-09-16T13:19:38.773578+0000","last_peered":"2024-09-16T13:19:38.773578+0000","last_clean":"2024-09-16T13:19:38.773578+0000","last_became_active":"2024-09-16T13:18:44.435188+0000","last_became_peered":"2024-09-16T13:18:44.435188+0000","last_unstale":"2024-09-16T13:19:38.773578+0000","last_undegraded":"2024-09-16T13:19:38.773578+0000","last_fullsized":"2024-09-16T13:19:38.773578+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:35.400164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"143'487","reported_seq":2228,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336304+0000","last_change":"2024-09-16T13:19:19.131189+0000","last_active":"2024-09-16T13:19:38.336304+0000","last_peered":"2024-09-16T13:19:38.336304+0000","last_clean":"2024-09-16T13:19:38.336304+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:19:38.336304+0000","last_undegraded":"2024-09-16T13:19:38.336304+0000","last_fullsized":"2024-09-16T13:19:38.336304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:45.154984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003557705,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3512,"num_read_kb":15515,"num_write":504,"num_write_kb":6261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473138+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:19:38.473138+0000","last_peered":"2024-09-16T13:19:38.473138+0000","last_clean":"2024-09-16T13:19:38.473138+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:19:38.473138+0000","last_undegraded":"2024-09-16T13:19:38.473138+0000","last_fullsized":"2024-09-16T13:19:38.473138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":236,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773593+0000","last_change":"2024-09-16T13:18:44.437802+0000","last_active":"2024-09-16T13:19:38.773593+0000","last_peered":"2024-09-16T13:19:38.773593+0000","last_clean":"2024-09-16T13:19:38.773593+0000","last_became_active":"2024-09-16T13:18:44.437418+0000","last_became_peered":"2024-09-16T13:18:44.437418+0000","last_unstale":"2024-09-16T13:19:38.773593+0000","last_undegraded":"2024-09-16T13:19:38.773593+0000","last_fullsized":"2024-09-16T13:19:38.773593+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:29.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"143'606","reported_seq":1309,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336209+0000","last_change":"2024-09-16T13:19:19.131515+0000","last_active":"2024-09-16T13:19:38.336209+0000","last_peered":"2024-09-16T13:19:38.336209+0000","last_clean":"2024-09-16T13:19:38.336209+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:19:38.336209+0000","last_undegraded":"2024-09-16T13:19:38.336209+0000","last_fullsized":"2024-09-16T13:19:38.336209+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:59.515115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033814320000000002,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2017,"num_read_kb":19372,"num_write":789,"num_write_kb":12171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"142'523","reported_seq":3009,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338761+0000","last_change":"2024-09-16T13:19:19.128362+0000","last_active":"2024-09-16T13:19:38.338761+0000","last_peered":"2024-09-16T13:19:38.338761+0000","last_clean":"2024-09-16T13:19:38.338761+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:19:38.338761+0000","last_undegraded":"2024-09-16T13:19:38.338761+0000","last_fullsized":"2024-09-16T13:19:38.338761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:49:10.301550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001533006,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4183,"num_read_kb":31351,"num_write":445,"num_write_kb":12351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"142'942","reported_seq":3022,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.347467+0000","last_change":"2024-09-16T13:19:38.346887+0000","last_active":"2024-09-16T13:19:38.347467+0000","last_peered":"2024-09-16T13:19:38.347467+0000","last_clean":"2024-09-16T13:19:38.347467+0000","last_became_active":"2024-09-16T13:19:38.343200+0000","last_became_peered":"2024-09-16T13:19:38.343200+0000","last_unstale":"2024-09-16T13:19:38.347467+0000","last_undegraded":"2024-09-16T13:19:38.347467+0000","last_fullsized":"2024-09-16T13:19:38.347467+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:37.001790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025018559999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":4754,"num_read_kb":19969,"num_write":1275,"num_write_kb":12323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"142'1140","reported_seq":4459,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336696+0000","last_change":"2024-09-16T13:19:19.128947+0000","last_active":"2024-09-16T13:19:38.336696+0000","last_peered":"2024-09-16T13:19:38.336696+0000","last_clean":"2024-09-16T13:19:38.336696+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:19:38.336696+0000","last_undegraded":"2024-09-16T13:19:38.336696+0000","last_fullsized":"2024-09-16T13:19:38.336696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:20.684083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014023239999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":9870,"num_read_kb":27372,"num_write":1699,"num_write_kb":10648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"143'323","reported_seq":1457,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473160+0000","last_change":"2024-09-16T13:19:19.135617+0000","last_active":"2024-09-16T13:19:38.473160+0000","last_peered":"2024-09-16T13:19:38.473160+0000","last_clean":"2024-09-16T13:19:38.473160+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:19:38.473160+0000","last_undegraded":"2024-09-16T13:19:38.473160+0000","last_fullsized":"2024-09-16T13:19:38.473160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:39.937083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010610159999999999,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":1447,"num_read_kb":15039,"num_write":293,"num_write_kb":7034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'389","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774008+0000","last_change":"2024-09-16T13:19:19.175390+0000","last_active":"2024-09-16T13:19:38.774008+0000","last_peered":"2024-09-16T13:19:38.774008+0000","last_clean":"2024-09-16T13:19:38.774008+0000","last_became_active":"2024-09-16T13:18:47.468001+0000","last_became_peered":"2024-09-16T13:18:47.468001+0000","last_unstale":"2024-09-16T13:19:38.774008+0000","last_undegraded":"2024-09-16T13:19:38.774008+0000","last_fullsized":"2024-09-16T13:19:38.774008+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:11:32.886561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018957069999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2747,"num_read_kb":18459,"num_write":343,"num_write_kb":14196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"142'510","reported_seq":7096,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773857+0000","last_change":"2024-09-16T13:19:19.173823+0000","last_active":"2024-09-16T13:19:38.773857+0000","last_peered":"2024-09-16T13:19:38.773857+0000","last_clean":"2024-09-16T13:19:38.773857+0000","last_became_active":"2024-09-16T13:18:47.467028+0000","last_became_peered":"2024-09-16T13:18:47.467028+0000","last_unstale":"2024-09-16T13:19:38.773857+0000","last_undegraded":"2024-09-16T13:19:38.773857+0000","last_fullsized":"2024-09-16T13:19:38.773857+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:35.852378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092761899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":8104,"num_read_kb":29095,"num_write":570,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1715710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"142'310","reported_seq":1005,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774295+0000","last_change":"2024-09-16T13:19:19.173006+0000","last_active":"2024-09-16T13:19:38.774295+0000","last_peered":"2024-09-16T13:19:38.774295+0000","last_clean":"2024-09-16T13:19:38.774295+0000","last_became_active":"2024-09-16T13:18:44.444025+0000","last_became_peered":"2024-09-16T13:18:44.444025+0000","last_unstale":"2024-09-16T13:19:38.774295+0000","last_undegraded":"2024-09-16T13:19:38.774295+0000","last_fullsized":"2024-09-16T13:19:38.774295+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:25:03.521529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056513400000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":692,"num_read_kb":10388,"num_write":284,"num_write_kb":7715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":63719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"142'265","reported_seq":1394,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.346281+0000","last_change":"2024-09-16T13:19:38.345793+0000","last_active":"2024-09-16T13:19:38.346281+0000","last_peered":"2024-09-16T13:19:38.346281+0000","last_clean":"2024-09-16T13:19:38.346281+0000","last_became_active":"2024-09-16T13:19:38.344229+0000","last_became_peered":"2024-09-16T13:19:38.344229+0000","last_unstale":"2024-09-16T13:19:38.346281+0000","last_undegraded":"2024-09-16T13:19:38.346281+0000","last_fullsized":"2024-09-16T13:19:38.346281+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:30:56.026666+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00077869099999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":784,"num_read_kb":16305,"num_write":257,"num_write_kb":11082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"142'556","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773502+0000","last_change":"2024-09-16T13:19:19.175226+0000","last_active":"2024-09-16T13:19:38.773502+0000","last_peered":"2024-09-16T13:19:38.773502+0000","last_clean":"2024-09-16T13:19:38.773502+0000","last_became_active":"2024-09-16T13:18:47.462561+0000","last_became_peered":"2024-09-16T13:18:47.462561+0000","last_unstale":"2024-09-16T13:19:38.773502+0000","last_undegraded":"2024-09-16T13:19:38.773502+0000","last_fullsized":"2024-09-16T13:19:38.773502+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.079652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021368229999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3858,"num_read_kb":23843,"num_write":732,"num_write_kb":11756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"141'398","reported_seq":7509,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343225+0000","last_change":"2024-09-16T13:19:19.164016+0000","last_active":"2024-09-16T13:19:38.343225+0000","last_peered":"2024-09-16T13:19:38.343225+0000","last_clean":"2024-09-16T13:19:38.343225+0000","last_became_active":"2024-09-16T13:18:47.462412+0000","last_became_peered":"2024-09-16T13:18:47.462412+0000","last_unstale":"2024-09-16T13:19:38.343225+0000","last_undegraded":"2024-09-16T13:19:38.343225+0000","last_fullsized":"2024-09-16T13:19:38.343225+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:50:25.256657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001704484,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":8501,"num_read_kb":35036,"num_write":362,"num_write_kb":9220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2494648,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"141'506","reported_seq":2090,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773579+0000","last_change":"2024-09-16T13:19:19.172960+0000","last_active":"2024-09-16T13:19:38.773579+0000","last_peered":"2024-09-16T13:19:38.773579+0000","last_clean":"2024-09-16T13:19:38.773579+0000","last_became_active":"2024-09-16T13:18:44.443832+0000","last_became_peered":"2024-09-16T13:18:44.443832+0000","last_unstale":"2024-09-16T13:19:38.773579+0000","last_undegraded":"2024-09-16T13:19:38.773579+0000","last_fullsized":"2024-09-16T13:19:38.773579+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:45.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001469344,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":5023,"num_read_kb":18731,"num_write":523,"num_write_kb":7705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1653183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":218,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":214,"num_read":143401,"num_read_kb":680525,"num_write":19167,"num_write_kb":295695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":44660033,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0},"log_size":16471,"ondisk_log_size":16471,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509482,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311724,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060116,"statfs":{"total":96636764160,"available":96317558784,"internally_reserved":0,"allocated":1929216,"data_stored":921154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.84999999999999998},"min":{"1min":0.47299999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2809999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.499,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1910000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":3.2130000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.83799999999999997,"15min":0.82799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.3009999999999999,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.98799999999999999,"15min":0.98199999999999998},"min":{"1min":0.64500000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.6789999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":1.1060000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.878,"15min":0.89400000000000002},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.268,"5min":1.476,"15min":1.476},"last":1.319},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.97699999999999998,"15min":1.0389999999999999},"min":{"1min":0.63300000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.278,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":3.1299999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.0409999999999999,"15min":1.081},"min":{"1min":0.73099999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.5350000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":1.4119999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.98599999999999999,"15min":0.999},"min":{"1min":0.65200000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.256,"5min":3.0209999999999999,"15min":3.0209999999999999},"last":2.9860000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.96599999999999997,"15min":0.98799999999999999},"min":{"1min":0.53800000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.335,"5min":1.825,"15min":1.825},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0309999999999999,"15min":1.0600000000000001},"min":{"1min":0.56299999999999994,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.347,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.004}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88100000000000001,"15min":0.90300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.409,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.115,"5min":1.1100000000000001,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":1.1559999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.24},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":3.286}]}]},{"osd":4,"up_from":15,"seq":64424509481,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311740,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060100,"statfs":{"total":96636764160,"available":96317542400,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.78300000000000003,"15min":0.80100000000000005},"min":{"1min":0.51600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.927,"15min":1.927},"last":1.282}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78700000000000003,"15min":0.78300000000000003},"min":{"1min":0.52300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.90000000000000002,"15min":0.91600000000000004},"min":{"1min":0.55000000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.3109999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.82499999999999996,"15min":0.82199999999999995},"min":{"1min":0.59699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.93500000000000005},"min":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0209999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.83599999999999997,"15min":0.84399999999999997},"min":{"1min":0.69699999999999995,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.167},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.94699999999999995,"15min":0.95399999999999996},"min":{"1min":0.85999999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.1339999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.96199999999999997},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.4239999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.341,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94599999999999995,"15min":0.97999999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.0609999999999999,"15min":1.0920000000000001},"min":{"1min":0.59899999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.369}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.2490000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509482,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311736,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060104,"statfs":{"total":96636764160,"available":96317546496,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.45400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.59499999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65000000000000002,"15min":0.66300000000000003},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92600000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.76600000000000001},"min":{"1min":0.48399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0660000000000001,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.76300000000000001,"15min":0.77100000000000002},"min":{"1min":0.57399999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.88500000000000001},"min":{"1min":0.56399999999999995,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1879999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.98599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0209999999999999,"15min":0.98899999999999999},"min":{"1min":0.747,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.7989999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":1.177},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.98999999999999999,"15min":0.96499999999999997},"min":{"1min":0.77500000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3100000000000001,"5min":4.9489999999999998,"15min":4.9489999999999998},"last":1.3100000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.99299999999999999,"15min":0.95599999999999996},"min":{"1min":0.71499999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3380000000000001,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":1.147},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.087,"15min":1.091},"min":{"1min":0.86499999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.45,"15min":1.45},"last":1.2849999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0649999999999999,"15min":1.0629999999999999},"min":{"1min":0.69999999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.843,"5min":2.843,"15min":2.843},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.01,"15min":1.01},"min":{"1min":0.80300000000000005,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.385}]},{"osd":7,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.2490000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509482,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303428,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068412,"statfs":{"total":96636764160,"available":96326053888,"internally_reserved":0,"allocated":1949696,"data_stored":927051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.911,"5min":3.911,"15min":3.911},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.46600000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9729999999999999,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":0.96099999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9510000000000001,"5min":2.9510000000000001,"15min":2.9510000000000001},"last":0.88}]},{"osd":3,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.75800000000000001,"15min":0.72799999999999998},"min":{"1min":0.58199999999999996,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.2089999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":0.80500000000000005,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.76500000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.86899999999999999,"15min":0.84399999999999997},"min":{"1min":0.56299999999999994,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.014},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.96799999999999997,"15min":0.96699999999999997},"min":{"1min":0.51500000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.343}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":0.97499999999999998,"15min":0.94099999999999995},"min":{"1min":0.86499999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":5.0919999999999996,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":1.101},{"interface":"front","average":{"1min":1.163,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.61899999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":3.8279999999999998},"last":0.94999999999999996}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.0429999999999999,"15min":1.0089999999999999},"min":{"1min":0.75700000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.04,"15min":1.0269999999999999},"min":{"1min":0.58299999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":4.0460000000000003,"5min":4.0460000000000003,"15min":4.0460000000000003},"last":1.085}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.995},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.288}]}]},{"osd":7,"up_from":107,"seq":459561500686,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305528,"kb_used_data":1960,"kb_used_omap":50,"kb_used_meta":303501,"kb_avail":94066312,"statfs":{"total":96636764160,"available":96323903488,"internally_reserved":0,"allocated":2007040,"data_stored":1517381,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":51722,"internal_metadata":310785526},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299264,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072576,"statfs":{"total":96636764160,"available":96330317824,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.77800000000000002,"15min":0.76600000000000001},"min":{"1min":0.57399999999999995,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2929999999999999,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96999999999999997,"15min":0.94799999999999995},"min":{"1min":0.52300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.335,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":1.016}]},{"osd":1,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.78500000000000003,"15min":0.77300000000000002},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86899999999999999,"15min":0.86499999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.81899999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84599999999999997,"15min":0.86599999999999999},"min":{"1min":0.60799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.1579999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.044,"5min":0.95999999999999996,"15min":0.95899999999999996},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3979999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.88200000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.97599999999999998,"15min":0.95799999999999996},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.107}]},{"osd":4,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.871,"15min":0.88700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1220000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.874,"5min":1.002,"15min":1.0229999999999999},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3009999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.2509999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.96599999999999997,"15min":0.97799999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.329,"5min":1.637,"15min":1.637},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0529999999999999,"15min":1.0720000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.1910000000000001,"5min":1.661,"15min":1.661},"last":1.1950000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.171},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.224}]}]},{"osd":2,"up_from":14,"seq":60129542185,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307580,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064260,"statfs":{"total":96636764160,"available":96321802240,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59699999999999998,"15min":0.60299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85599999999999998,"5min":1.159,"15min":1.159},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.38600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.506,"15min":1.506},"last":0.33400000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.438,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.05,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.218,"5min":3.298,"15min":3.298},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79600000000000004,"15min":0.80700000000000005},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0980000000000001,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.91800000000000004},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.266,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.97999999999999998,"15min":1.014},"min":{"1min":0.61499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.5780000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.05,"5min":1.002,"15min":1},"min":{"1min":0.57299999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.6279999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.92600000000000005}]},{"osd":5,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0569999999999999,"15min":1.081},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.726,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.131,"5min":1.0880000000000001,"15min":1.1060000000000001},"min":{"1min":0.69799999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5029999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.131}]},{"osd":6,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.054,"15min":1.069},"min":{"1min":0.67600000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.758,"5min":1.857,"15min":1.857},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.141,"15min":1.123},"min":{"1min":0.64600000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4079999999999999,"5min":7.0090000000000003,"15min":7.0090000000000003},"last":1.1779999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.056},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":0.63800000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303464,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068376,"statfs":{"total":96636764160,"available":96326017024,"internally_reserved":0,"allocated":1990656,"data_stored":1517381,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.60699999999999998,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.501,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.01,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.70099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.224,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.53400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.109,"5min":1.74,"15min":1.74},"last":0.91100000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.76200000000000001,"15min":0.77600000000000002},"min":{"1min":0.50900000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.48099999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.276,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1}]},{"osd":4,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.92700000000000005,"15min":0.92500000000000004},"min":{"1min":0.73399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.615,"5min":1.871,"15min":1.871},"last":1.048},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.71499999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.2809999999999999,"5min":1.903,"15min":1.903},"last":0.86199999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0269999999999999,"15min":1.0069999999999999},"min":{"1min":0.80100000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.357,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.242},{"interface":"front","average":{"1min":1.073,"5min":0.97999999999999998,"15min":0.96299999999999997},"min":{"1min":0.70099999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.103}]},{"osd":6,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.044,"15min":1.0409999999999999},"min":{"1min":0.71299999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3879999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0369999999999999,"15min":1.0269999999999999},"min":{"1min":0.75600000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5149999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.0740000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.881,"5min":1.881,"15min":1.881},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0}]}} 2024-09-16T13:19:44.301 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T13:19:44.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-16T13:19:44.597 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T13:19:44.605 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-16T13:19:44.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:19:44.879 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:44.879 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":162,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:19:39.331549+0000","last_up_change":"2024-09-16T13:18:43.413345+0000","last_in_change":"2024-09-16T13:17:42.548353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:16:33.497561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:16:40.994538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-16T13:16:50.759705+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":81,"snap_epoch":143,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":22271},{"type":"v1","addr":"172.21.15.98:6809","nonce":22271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":22271},{"type":"v1","addr":"172.21.15.98:6811","nonce":22271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":22271},{"type":"v1","addr":"172.21.15.98:6817","nonce":22271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":22271},{"type":"v1","addr":"172.21.15.98:6814","nonce":22271}]},"public_addr":"172.21.15.98:6809/22271","cluster_addr":"172.21.15.98:6811/22271","heartbeat_back_addr":"172.21.15.98:6817/22271","heartbeat_front_addr":"172.21.15.98:6814/22271","state":["exists","up"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":22272},{"type":"v1","addr":"172.21.15.98:6815","nonce":22272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":22272},{"type":"v1","addr":"172.21.15.98:6819","nonce":22272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":22272},{"type":"v1","addr":"172.21.15.98:6823","nonce":22272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":22272},{"type":"v1","addr":"172.21.15.98:6821","nonce":22272}]},"public_addr":"172.21.15.98:6815/22272","cluster_addr":"172.21.15.98:6819/22272","heartbeat_back_addr":"172.21.15.98:6823/22272","heartbeat_front_addr":"172.21.15.98:6821/22272","state":["exists","up"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":22273},{"type":"v1","addr":"172.21.15.98:6801","nonce":22273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":22273},{"type":"v1","addr":"172.21.15.98:6803","nonce":22273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":22273},{"type":"v1","addr":"172.21.15.98:6807","nonce":22273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":22273},{"type":"v1","addr":"172.21.15.98:6805","nonce":22273}]},"public_addr":"172.21.15.98:6801/22273","cluster_addr":"172.21.15.98:6803/22273","heartbeat_back_addr":"172.21.15.98:6807/22273","heartbeat_front_addr":"172.21.15.98:6805/22273","state":["exists","up"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":22270},{"type":"v1","addr":"172.21.15.98:6825","nonce":22270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":22270},{"type":"v1","addr":"172.21.15.98:6827","nonce":22270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":22270},{"type":"v1","addr":"172.21.15.98:6831","nonce":22270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":22270},{"type":"v1","addr":"172.21.15.98:6829","nonce":22270}]},"public_addr":"172.21.15.98:6825/22270","cluster_addr":"172.21.15.98:6827/22270","heartbeat_back_addr":"172.21.15.98:6831/22270","heartbeat_front_addr":"172.21.15.98:6829/22270","state":["exists","up"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":22286},{"type":"v1","addr":"172.21.15.151:6825","nonce":22286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":22286},{"type":"v1","addr":"172.21.15.151:6827","nonce":22286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":22286},{"type":"v1","addr":"172.21.15.151:6831","nonce":22286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":22286},{"type":"v1","addr":"172.21.15.151:6829","nonce":22286}]},"public_addr":"172.21.15.151:6825/22286","cluster_addr":"172.21.15.151:6827/22286","heartbeat_back_addr":"172.21.15.151:6831/22286","heartbeat_front_addr":"172.21.15.151:6829/22286","state":["exists","up"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":22285},{"type":"v1","addr":"172.21.15.151:6809","nonce":22285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":22285},{"type":"v1","addr":"172.21.15.151:6811","nonce":22285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":22285},{"type":"v1","addr":"172.21.15.151:6815","nonce":22285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":22285},{"type":"v1","addr":"172.21.15.151:6813","nonce":22285}]},"public_addr":"172.21.15.151:6809/22285","cluster_addr":"172.21.15.151:6811/22285","heartbeat_back_addr":"172.21.15.151:6815/22285","heartbeat_front_addr":"172.21.15.151:6813/22285","state":["exists","up"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":22315},{"type":"v1","addr":"172.21.15.151:6817","nonce":22315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":22315},{"type":"v1","addr":"172.21.15.151:6819","nonce":22315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":22315},{"type":"v1","addr":"172.21.15.151:6823","nonce":22315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":22315},{"type":"v1","addr":"172.21.15.151:6821","nonce":22315}]},"public_addr":"172.21.15.151:6817/22315","cluster_addr":"172.21.15.151:6819/22315","heartbeat_back_addr":"172.21.15.151:6823/22315","heartbeat_front_addr":"172.21.15.151:6821/22315","state":["exists","up"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":81,"up_from":107,"up_thru":137,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":33053},{"type":"v1","addr":"172.21.15.151:6801","nonce":33053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":33053},{"type":"v1","addr":"172.21.15.151:6803","nonce":33053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":33053},{"type":"v1","addr":"172.21.15.151:6807","nonce":33053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":33053},{"type":"v1","addr":"172.21.15.151:6805","nonce":33053}]},"public_addr":"172.21.15.151:6801/33053","cluster_addr":"172.21.15.151:6803/33053","heartbeat_back_addr":"172.21.15.151:6807/33053","heartbeat_front_addr":"172.21.15.151:6805/33053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.191311+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.563551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.493457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.599815+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.634352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.262873+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.508881+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.348276+0000","dead_epoch":82}],"pg_upmap":[{"pgid":"3.1c","osds":[6,1]},{"pgid":"3.1f","osds":[5,3]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:19:44.900 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T13:19:44.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:19:45.199 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:45.203 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:19:45.216 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-16T13:19:43.492206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":214,"num_read":143467,"num_read_kb":680580,"num_write":19270,"num_write_kb":297325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":45250401,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16546,"ondisk_log_size":16546,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2454464,"kb_used_data":15284,"kb_used_omap":62,"kb_used_meta":2438977,"kb_avail":752520256,"statfs":{"total":773094113280,"available":770580742144,"internally_reserved":0,"allocated":15650816,"data_stored":8591057,"data_compressed":67468,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":63650,"internal_metadata":2497513310},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.157128"},"pg_stats":[{"pgid":"3.1f","version":"143'370","reported_seq":1980,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473500+0000","last_change":"2024-09-16T13:19:19.135941+0000","last_active":"2024-09-16T13:19:38.473500+0000","last_peered":"2024-09-16T13:19:38.473500+0000","last_clean":"2024-09-16T13:19:38.473500+0000","last_became_active":"2024-09-16T13:18:03.119735+0000","last_became_peered":"2024-09-16T13:18:03.119735+0000","last_unstale":"2024-09-16T13:19:38.473500+0000","last_undegraded":"2024-09-16T13:19:38.473500+0000","last_fullsized":"2024-09-16T13:19:38.473500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:02.061478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012744049999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2248,"num_read_kb":16063,"num_write":361,"num_write_kb":5586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":922127,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"141'824","reported_seq":2700,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336528+0000","last_change":"2024-09-16T13:19:19.131668+0000","last_active":"2024-09-16T13:19:38.336528+0000","last_peered":"2024-09-16T13:19:38.336528+0000","last_clean":"2024-09-16T13:19:38.336528+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:19:38.336528+0000","last_undegraded":"2024-09-16T13:19:38.336528+0000","last_fullsized":"2024-09-16T13:19:38.336528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:50.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003379118,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":5839,"num_read_kb":7327,"num_write":1002,"num_write_kb":6247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"141'446","reported_seq":2670,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473459+0000","last_change":"2024-09-16T13:19:19.136172+0000","last_active":"2024-09-16T13:19:38.473459+0000","last_peered":"2024-09-16T13:19:38.473459+0000","last_clean":"2024-09-16T13:19:38.473459+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:19:38.473459+0000","last_undegraded":"2024-09-16T13:19:38.473459+0000","last_fullsized":"2024-09-16T13:19:38.473459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:14.387827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014771560000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4778,"num_read_kb":13874,"num_write":452,"num_write_kb":6042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"143'449","reported_seq":2727,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.344857+0000","last_change":"2024-09-16T13:19:38.344285+0000","last_active":"2024-09-16T13:19:38.344857+0000","last_peered":"2024-09-16T13:19:38.344857+0000","last_clean":"2024-09-16T13:19:38.344857+0000","last_became_active":"2024-09-16T13:19:38.342560+0000","last_became_peered":"2024-09-16T13:19:38.342560+0000","last_unstale":"2024-09-16T13:19:38.344857+0000","last_undegraded":"2024-09-16T13:19:38.344857+0000","last_fullsized":"2024-09-16T13:19:38.344857+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:57:32.494600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00301937,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":2995,"num_read_kb":25210,"num_write":556,"num_write_kb":12813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"141'671","reported_seq":5617,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343768+0000","last_change":"2024-09-16T13:19:19.164598+0000","last_active":"2024-09-16T13:19:38.343768+0000","last_peered":"2024-09-16T13:19:38.343768+0000","last_clean":"2024-09-16T13:19:38.343768+0000","last_became_active":"2024-09-16T13:19:10.071308+0000","last_became_peered":"2024-09-16T13:19:10.071308+0000","last_unstale":"2024-09-16T13:19:38.343768+0000","last_undegraded":"2024-09-16T13:19:38.343768+0000","last_fullsized":"2024-09-16T13:19:38.343768+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:10:31.562928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002223587,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6572,"num_read_kb":28113,"num_write":920,"num_write_kb":17343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":3647407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"142'661","reported_seq":7962,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338599+0000","last_change":"2024-09-16T13:19:19.129707+0000","last_active":"2024-09-16T13:19:38.338599+0000","last_peered":"2024-09-16T13:19:38.338599+0000","last_clean":"2024-09-16T13:19:38.338599+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:19:38.338599+0000","last_undegraded":"2024-09-16T13:19:38.338599+0000","last_fullsized":"2024-09-16T13:19:38.338599+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:31.071219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028974579999999999,"stat_sum":{"num_bytes":19,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":9256,"num_read_kb":55053,"num_write":857,"num_write_kb":18796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"142'608","reported_seq":5503,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.335998+0000","last_change":"2024-09-16T13:19:19.128105+0000","last_active":"2024-09-16T13:19:38.335998+0000","last_peered":"2024-09-16T13:19:38.335998+0000","last_clean":"2024-09-16T13:19:38.335998+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:19:38.335998+0000","last_undegraded":"2024-09-16T13:19:38.335998+0000","last_fullsized":"2024-09-16T13:19:38.335998+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:42:30.327300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002453879,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":7637,"num_read_kb":32040,"num_write":812,"num_write_kb":13589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'421","reported_seq":2587,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336098+0000","last_change":"2024-09-16T13:19:19.131005+0000","last_active":"2024-09-16T13:19:38.336098+0000","last_peered":"2024-09-16T13:19:38.336098+0000","last_clean":"2024-09-16T13:19:38.336098+0000","last_became_active":"2024-09-16T13:19:10.071112+0000","last_became_peered":"2024-09-16T13:19:10.071112+0000","last_unstale":"2024-09-16T13:19:38.336098+0000","last_undegraded":"2024-09-16T13:19:38.336098+0000","last_fullsized":"2024-09-16T13:19:38.336098+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:29.996047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032828369999999998,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2803,"num_read_kb":8703,"num_write":538,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":228967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"141'573","reported_seq":2387,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343524+0000","last_change":"2024-09-16T13:19:19.163188+0000","last_active":"2024-09-16T13:19:38.343524+0000","last_peered":"2024-09-16T13:19:38.343524+0000","last_clean":"2024-09-16T13:19:38.343524+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:19:38.343524+0000","last_undegraded":"2024-09-16T13:19:38.343524+0000","last_fullsized":"2024-09-16T13:19:38.343524+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:08:57.225261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000869449,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3700,"num_read_kb":13824,"num_write":600,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'410","reported_seq":3104,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473297+0000","last_change":"2024-09-16T13:19:19.135414+0000","last_active":"2024-09-16T13:19:38.473297+0000","last_peered":"2024-09-16T13:19:38.473297+0000","last_clean":"2024-09-16T13:19:38.473297+0000","last_became_active":"2024-09-16T13:18:44.441740+0000","last_became_peered":"2024-09-16T13:18:44.441740+0000","last_unstale":"2024-09-16T13:19:38.473297+0000","last_undegraded":"2024-09-16T13:19:38.473297+0000","last_fullsized":"2024-09-16T13:19:38.473297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:20.994894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082856900000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":5473,"num_read_kb":21309,"num_write":357,"num_write_kb":5655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1185011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"143'229","reported_seq":831,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773785+0000","last_change":"2024-09-16T13:19:19.174836+0000","last_active":"2024-09-16T13:19:38.773785+0000","last_peered":"2024-09-16T13:19:38.773785+0000","last_clean":"2024-09-16T13:19:38.773785+0000","last_became_active":"2024-09-16T13:18:44.435407+0000","last_became_peered":"2024-09-16T13:18:44.435407+0000","last_unstale":"2024-09-16T13:19:38.773785+0000","last_undegraded":"2024-09-16T13:19:38.773785+0000","last_fullsized":"2024-09-16T13:19:38.773785+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:11.546854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016773129999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3249,"num_read_kb":15295,"num_write":190,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1086312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"142'304","reported_seq":5066,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338671+0000","last_change":"2024-09-16T13:19:19.128958+0000","last_active":"2024-09-16T13:19:38.338671+0000","last_peered":"2024-09-16T13:19:38.338671+0000","last_clean":"2024-09-16T13:19:38.338671+0000","last_became_active":"2024-09-16T13:18:47.462532+0000","last_became_peered":"2024-09-16T13:18:47.462532+0000","last_unstale":"2024-09-16T13:19:38.338671+0000","last_undegraded":"2024-09-16T13:19:38.338671+0000","last_fullsized":"2024-09-16T13:19:38.338671+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:16.599784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002111417,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4736,"num_read_kb":25385,"num_write":347,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1053570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.777150+0000","last_change":"2024-09-16T13:19:38.776183+0000","last_active":"2024-09-16T13:19:38.777150+0000","last_peered":"2024-09-16T13:19:38.777150+0000","last_clean":"2024-09-16T13:19:38.777150+0000","last_became_active":"2024-09-16T13:19:38.775280+0000","last_became_peered":"2024-09-16T13:19:38.775280+0000","last_unstale":"2024-09-16T13:19:38.777150+0000","last_undegraded":"2024-09-16T13:19:38.777150+0000","last_fullsized":"2024-09-16T13:19:38.777150+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:46.699452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"143'692","reported_seq":5654,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343415+0000","last_change":"2024-09-16T13:19:19.164842+0000","last_active":"2024-09-16T13:19:38.343415+0000","last_peered":"2024-09-16T13:19:38.343415+0000","last_clean":"2024-09-16T13:19:38.343415+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:19:38.343415+0000","last_undegraded":"2024-09-16T13:19:38.343415+0000","last_fullsized":"2024-09-16T13:19:38.343415+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:52:05.144800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022584039999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6654,"num_read_kb":24625,"num_write":939,"num_write_kb":11031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336150+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:19:38.336150+0000","last_peered":"2024-09-16T13:19:38.336150+0000","last_clean":"2024-09-16T13:19:38.336150+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:19:38.336150+0000","last_undegraded":"2024-09-16T13:19:38.336150+0000","last_fullsized":"2024-09-16T13:19:38.336150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"142'303","reported_seq":1472,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336544+0000","last_change":"2024-09-16T13:19:19.132140+0000","last_active":"2024-09-16T13:19:38.336544+0000","last_peered":"2024-09-16T13:19:38.336544+0000","last_clean":"2024-09-16T13:19:38.336544+0000","last_became_active":"2024-09-16T13:18:47.463039+0000","last_became_peered":"2024-09-16T13:18:47.463039+0000","last_unstale":"2024-09-16T13:19:38.336544+0000","last_undegraded":"2024-09-16T13:19:38.336544+0000","last_fullsized":"2024-09-16T13:19:38.336544+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:05:53.068034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026327310000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":1655,"num_read_kb":8730,"num_write":270,"num_write_kb":5747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1061466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":136,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773757+0000","last_change":"2024-09-16T13:18:44.435471+0000","last_active":"2024-09-16T13:19:38.773757+0000","last_peered":"2024-09-16T13:19:38.773757+0000","last_clean":"2024-09-16T13:19:38.773757+0000","last_became_active":"2024-09-16T13:18:44.434851+0000","last_became_peered":"2024-09-16T13:18:44.434851+0000","last_unstale":"2024-09-16T13:19:38.773757+0000","last_undegraded":"2024-09-16T13:19:38.773757+0000","last_fullsized":"2024-09-16T13:19:38.773757+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:12.980758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"141'372","reported_seq":4687,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336707+0000","last_change":"2024-09-16T13:19:19.129830+0000","last_active":"2024-09-16T13:19:38.336707+0000","last_peered":"2024-09-16T13:19:38.336707+0000","last_clean":"2024-09-16T13:19:38.336707+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:19:38.336707+0000","last_undegraded":"2024-09-16T13:19:38.336707+0000","last_fullsized":"2024-09-16T13:19:38.336707+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:30.996264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032315740000000001,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4693,"num_read_kb":26876,"num_write":422,"num_write_kb":8111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":172,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336904+0000","last_change":"2024-09-16T13:17:42.960492+0000","last_active":"2024-09-16T13:19:38.336904+0000","last_peered":"2024-09-16T13:19:38.336904+0000","last_clean":"2024-09-16T13:19:38.336904+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:19:38.336904+0000","last_undegraded":"2024-09-16T13:19:38.336904+0000","last_fullsized":"2024-09-16T13:19:38.336904+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:35:45.152557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"143'544","reported_seq":3842,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473442+0000","last_change":"2024-09-16T13:19:19.136757+0000","last_active":"2024-09-16T13:19:38.473442+0000","last_peered":"2024-09-16T13:19:38.473442+0000","last_clean":"2024-09-16T13:19:38.473442+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:19:38.473442+0000","last_undegraded":"2024-09-16T13:19:38.473442+0000","last_fullsized":"2024-09-16T13:19:38.473442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:34.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789768,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5947,"num_read_kb":14133,"num_write":543,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"143'506","reported_seq":2497,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.339019+0000","last_change":"2024-09-16T13:19:19.130409+0000","last_active":"2024-09-16T13:19:38.339019+0000","last_peered":"2024-09-16T13:19:38.339019+0000","last_clean":"2024-09-16T13:19:38.339019+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:19:38.339019+0000","last_undegraded":"2024-09-16T13:19:38.339019+0000","last_fullsized":"2024-09-16T13:19:38.339019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:44:16.402879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003357956,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2667,"num_read_kb":21275,"num_write":487,"num_write_kb":9028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":156,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473170+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:19:38.473170+0000","last_peered":"2024-09-16T13:19:38.473170+0000","last_clean":"2024-09-16T13:19:38.473170+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:19:38.473170+0000","last_undegraded":"2024-09-16T13:19:38.473170+0000","last_fullsized":"2024-09-16T13:19:38.473170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"142'424","reported_seq":1787,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.335712+0000","last_change":"2024-09-16T13:19:19.127890+0000","last_active":"2024-09-16T13:19:38.335712+0000","last_peered":"2024-09-16T13:19:38.335712+0000","last_clean":"2024-09-16T13:19:38.335712+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:19:38.335712+0000","last_undegraded":"2024-09-16T13:19:38.335712+0000","last_fullsized":"2024-09-16T13:19:38.335712+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:29.090161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020916900000000002,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3126,"num_read_kb":11030,"num_write":475,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773196+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:19:38.773196+0000","last_peered":"2024-09-16T13:19:38.773196+0000","last_clean":"2024-09-16T13:19:38.773196+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:19:38.773196+0000","last_undegraded":"2024-09-16T13:19:38.773196+0000","last_fullsized":"2024-09-16T13:19:38.773196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'709","reported_seq":2213,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343225+0000","last_change":"2024-09-16T13:19:19.164008+0000","last_active":"2024-09-16T13:19:38.343225+0000","last_peered":"2024-09-16T13:19:38.343225+0000","last_clean":"2024-09-16T13:19:38.343225+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:19:38.343225+0000","last_undegraded":"2024-09-16T13:19:38.343225+0000","last_fullsized":"2024-09-16T13:19:38.343225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:03.357310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017855869999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3881,"num_read_kb":31185,"num_write":963,"num_write_kb":16374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773578+0000","last_change":"2024-09-16T13:18:44.435672+0000","last_active":"2024-09-16T13:19:38.773578+0000","last_peered":"2024-09-16T13:19:38.773578+0000","last_clean":"2024-09-16T13:19:38.773578+0000","last_became_active":"2024-09-16T13:18:44.435188+0000","last_became_peered":"2024-09-16T13:18:44.435188+0000","last_unstale":"2024-09-16T13:19:38.773578+0000","last_undegraded":"2024-09-16T13:19:38.773578+0000","last_fullsized":"2024-09-16T13:19:38.773578+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:35.400164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"143'487","reported_seq":2228,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336304+0000","last_change":"2024-09-16T13:19:19.131189+0000","last_active":"2024-09-16T13:19:38.336304+0000","last_peered":"2024-09-16T13:19:38.336304+0000","last_clean":"2024-09-16T13:19:38.336304+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:19:38.336304+0000","last_undegraded":"2024-09-16T13:19:38.336304+0000","last_fullsized":"2024-09-16T13:19:38.336304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:45.154984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003557705,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3512,"num_read_kb":15515,"num_write":504,"num_write_kb":6261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473138+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:19:38.473138+0000","last_peered":"2024-09-16T13:19:38.473138+0000","last_clean":"2024-09-16T13:19:38.473138+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:19:38.473138+0000","last_undegraded":"2024-09-16T13:19:38.473138+0000","last_fullsized":"2024-09-16T13:19:38.473138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":236,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773593+0000","last_change":"2024-09-16T13:18:44.437802+0000","last_active":"2024-09-16T13:19:38.773593+0000","last_peered":"2024-09-16T13:19:38.773593+0000","last_clean":"2024-09-16T13:19:38.773593+0000","last_became_active":"2024-09-16T13:18:44.437418+0000","last_became_peered":"2024-09-16T13:18:44.437418+0000","last_unstale":"2024-09-16T13:19:38.773593+0000","last_undegraded":"2024-09-16T13:19:38.773593+0000","last_fullsized":"2024-09-16T13:19:38.773593+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:29.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"143'606","reported_seq":1309,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336209+0000","last_change":"2024-09-16T13:19:19.131515+0000","last_active":"2024-09-16T13:19:38.336209+0000","last_peered":"2024-09-16T13:19:38.336209+0000","last_clean":"2024-09-16T13:19:38.336209+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:19:38.336209+0000","last_undegraded":"2024-09-16T13:19:38.336209+0000","last_fullsized":"2024-09-16T13:19:38.336209+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:59.515115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033814320000000002,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2017,"num_read_kb":19372,"num_write":789,"num_write_kb":12171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"142'523","reported_seq":3009,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338761+0000","last_change":"2024-09-16T13:19:19.128362+0000","last_active":"2024-09-16T13:19:38.338761+0000","last_peered":"2024-09-16T13:19:38.338761+0000","last_clean":"2024-09-16T13:19:38.338761+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:19:38.338761+0000","last_undegraded":"2024-09-16T13:19:38.338761+0000","last_fullsized":"2024-09-16T13:19:38.338761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:49:10.301550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001533006,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4183,"num_read_kb":31351,"num_write":445,"num_write_kb":12351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"142'942","reported_seq":3022,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.347467+0000","last_change":"2024-09-16T13:19:38.346887+0000","last_active":"2024-09-16T13:19:38.347467+0000","last_peered":"2024-09-16T13:19:38.347467+0000","last_clean":"2024-09-16T13:19:38.347467+0000","last_became_active":"2024-09-16T13:19:38.343200+0000","last_became_peered":"2024-09-16T13:19:38.343200+0000","last_unstale":"2024-09-16T13:19:38.347467+0000","last_undegraded":"2024-09-16T13:19:38.347467+0000","last_fullsized":"2024-09-16T13:19:38.347467+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:37.001790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025018559999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":4754,"num_read_kb":19969,"num_write":1275,"num_write_kb":12323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"142'1140","reported_seq":4459,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336696+0000","last_change":"2024-09-16T13:19:19.128947+0000","last_active":"2024-09-16T13:19:38.336696+0000","last_peered":"2024-09-16T13:19:38.336696+0000","last_clean":"2024-09-16T13:19:38.336696+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:19:38.336696+0000","last_undegraded":"2024-09-16T13:19:38.336696+0000","last_fullsized":"2024-09-16T13:19:38.336696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:20.684083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014023239999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":9870,"num_read_kb":27372,"num_write":1699,"num_write_kb":10648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"143'323","reported_seq":1457,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473160+0000","last_change":"2024-09-16T13:19:19.135617+0000","last_active":"2024-09-16T13:19:38.473160+0000","last_peered":"2024-09-16T13:19:38.473160+0000","last_clean":"2024-09-16T13:19:38.473160+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:19:38.473160+0000","last_undegraded":"2024-09-16T13:19:38.473160+0000","last_fullsized":"2024-09-16T13:19:38.473160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:39.937083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010610159999999999,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":1447,"num_read_kb":15039,"num_write":293,"num_write_kb":7034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'389","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774008+0000","last_change":"2024-09-16T13:19:19.175390+0000","last_active":"2024-09-16T13:19:38.774008+0000","last_peered":"2024-09-16T13:19:38.774008+0000","last_clean":"2024-09-16T13:19:38.774008+0000","last_became_active":"2024-09-16T13:18:47.468001+0000","last_became_peered":"2024-09-16T13:18:47.468001+0000","last_unstale":"2024-09-16T13:19:38.774008+0000","last_undegraded":"2024-09-16T13:19:38.774008+0000","last_fullsized":"2024-09-16T13:19:38.774008+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:11:32.886561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018957069999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2747,"num_read_kb":18459,"num_write":343,"num_write_kb":14196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"142'510","reported_seq":7096,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773857+0000","last_change":"2024-09-16T13:19:19.173823+0000","last_active":"2024-09-16T13:19:38.773857+0000","last_peered":"2024-09-16T13:19:38.773857+0000","last_clean":"2024-09-16T13:19:38.773857+0000","last_became_active":"2024-09-16T13:18:47.467028+0000","last_became_peered":"2024-09-16T13:18:47.467028+0000","last_unstale":"2024-09-16T13:19:38.773857+0000","last_undegraded":"2024-09-16T13:19:38.773857+0000","last_fullsized":"2024-09-16T13:19:38.773857+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:35.852378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092761899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":8104,"num_read_kb":29095,"num_write":570,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1715710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"142'310","reported_seq":1005,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774295+0000","last_change":"2024-09-16T13:19:19.173006+0000","last_active":"2024-09-16T13:19:38.774295+0000","last_peered":"2024-09-16T13:19:38.774295+0000","last_clean":"2024-09-16T13:19:38.774295+0000","last_became_active":"2024-09-16T13:18:44.444025+0000","last_became_peered":"2024-09-16T13:18:44.444025+0000","last_unstale":"2024-09-16T13:19:38.774295+0000","last_undegraded":"2024-09-16T13:19:38.774295+0000","last_fullsized":"2024-09-16T13:19:38.774295+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:25:03.521529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056513400000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":692,"num_read_kb":10388,"num_write":284,"num_write_kb":7715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":63719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"142'265","reported_seq":1394,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.346281+0000","last_change":"2024-09-16T13:19:38.345793+0000","last_active":"2024-09-16T13:19:38.346281+0000","last_peered":"2024-09-16T13:19:38.346281+0000","last_clean":"2024-09-16T13:19:38.346281+0000","last_became_active":"2024-09-16T13:19:38.344229+0000","last_became_peered":"2024-09-16T13:19:38.344229+0000","last_unstale":"2024-09-16T13:19:38.346281+0000","last_undegraded":"2024-09-16T13:19:38.346281+0000","last_fullsized":"2024-09-16T13:19:38.346281+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:30:56.026666+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00077869099999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":784,"num_read_kb":16305,"num_write":257,"num_write_kb":11082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"142'556","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773502+0000","last_change":"2024-09-16T13:19:19.175226+0000","last_active":"2024-09-16T13:19:38.773502+0000","last_peered":"2024-09-16T13:19:38.773502+0000","last_clean":"2024-09-16T13:19:38.773502+0000","last_became_active":"2024-09-16T13:18:47.462561+0000","last_became_peered":"2024-09-16T13:18:47.462561+0000","last_unstale":"2024-09-16T13:19:38.773502+0000","last_undegraded":"2024-09-16T13:19:38.773502+0000","last_fullsized":"2024-09-16T13:19:38.773502+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.079652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021368229999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3858,"num_read_kb":23843,"num_write":732,"num_write_kb":11756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"141'398","reported_seq":7509,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343225+0000","last_change":"2024-09-16T13:19:19.164016+0000","last_active":"2024-09-16T13:19:38.343225+0000","last_peered":"2024-09-16T13:19:38.343225+0000","last_clean":"2024-09-16T13:19:38.343225+0000","last_became_active":"2024-09-16T13:18:47.462412+0000","last_became_peered":"2024-09-16T13:18:47.462412+0000","last_unstale":"2024-09-16T13:19:38.343225+0000","last_undegraded":"2024-09-16T13:19:38.343225+0000","last_fullsized":"2024-09-16T13:19:38.343225+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:50:25.256657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001704484,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":8501,"num_read_kb":35036,"num_write":362,"num_write_kb":9220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2494648,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"141'506","reported_seq":2090,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773579+0000","last_change":"2024-09-16T13:19:19.172960+0000","last_active":"2024-09-16T13:19:38.773579+0000","last_peered":"2024-09-16T13:19:38.773579+0000","last_clean":"2024-09-16T13:19:38.773579+0000","last_became_active":"2024-09-16T13:18:44.443832+0000","last_became_peered":"2024-09-16T13:18:44.443832+0000","last_unstale":"2024-09-16T13:19:38.773579+0000","last_undegraded":"2024-09-16T13:19:38.773579+0000","last_fullsized":"2024-09-16T13:19:38.773579+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:45.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001469344,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":5023,"num_read_kb":18731,"num_write":523,"num_write_kb":7705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1653183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":218,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":214,"num_read":143401,"num_read_kb":680525,"num_write":19167,"num_write_kb":295695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":44660033,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0},"log_size":16471,"ondisk_log_size":16471,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509482,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311724,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060116,"statfs":{"total":96636764160,"available":96317558784,"internally_reserved":0,"allocated":1929216,"data_stored":921154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.84999999999999998},"min":{"1min":0.47299999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2809999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.499,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1910000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":3.2130000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.83799999999999997,"15min":0.82799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.3009999999999999,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.98799999999999999,"15min":0.98199999999999998},"min":{"1min":0.64500000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.6789999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":1.1060000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.878,"15min":0.89400000000000002},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.268,"5min":1.476,"15min":1.476},"last":1.319},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.97699999999999998,"15min":1.0389999999999999},"min":{"1min":0.63300000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.278,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":3.1299999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.0409999999999999,"15min":1.081},"min":{"1min":0.73099999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.5350000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":1.4119999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.98599999999999999,"15min":0.999},"min":{"1min":0.65200000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.256,"5min":3.0209999999999999,"15min":3.0209999999999999},"last":2.9860000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.96599999999999997,"15min":0.98799999999999999},"min":{"1min":0.53800000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.335,"5min":1.825,"15min":1.825},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0309999999999999,"15min":1.0600000000000001},"min":{"1min":0.56299999999999994,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.347,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.004}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88100000000000001,"15min":0.90300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.409,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.115,"5min":1.1100000000000001,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":1.1559999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.24},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":3.286}]}]},{"osd":4,"up_from":15,"seq":64424509481,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311740,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060100,"statfs":{"total":96636764160,"available":96317542400,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.78300000000000003,"15min":0.80100000000000005},"min":{"1min":0.51600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.927,"15min":1.927},"last":1.282}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78700000000000003,"15min":0.78300000000000003},"min":{"1min":0.52300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.90000000000000002,"15min":0.91600000000000004},"min":{"1min":0.55000000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.3109999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.82499999999999996,"15min":0.82199999999999995},"min":{"1min":0.59699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.93500000000000005},"min":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0209999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.83599999999999997,"15min":0.84399999999999997},"min":{"1min":0.69699999999999995,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.167},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.94699999999999995,"15min":0.95399999999999996},"min":{"1min":0.85999999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.1339999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.96199999999999997},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.4239999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.341,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94599999999999995,"15min":0.97999999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.0609999999999999,"15min":1.0920000000000001},"min":{"1min":0.59899999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.369}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.2490000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509482,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311736,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060104,"statfs":{"total":96636764160,"available":96317546496,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.45400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.59499999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65000000000000002,"15min":0.66300000000000003},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92600000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.76600000000000001},"min":{"1min":0.48399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0660000000000001,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.76300000000000001,"15min":0.77100000000000002},"min":{"1min":0.57399999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.88500000000000001},"min":{"1min":0.56399999999999995,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1879999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.98599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0209999999999999,"15min":0.98899999999999999},"min":{"1min":0.747,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.7989999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":1.177},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.98999999999999999,"15min":0.96499999999999997},"min":{"1min":0.77500000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3100000000000001,"5min":4.9489999999999998,"15min":4.9489999999999998},"last":1.3100000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.99299999999999999,"15min":0.95599999999999996},"min":{"1min":0.71499999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3380000000000001,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":1.147},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.087,"15min":1.091},"min":{"1min":0.86499999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.45,"15min":1.45},"last":1.2849999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0649999999999999,"15min":1.0629999999999999},"min":{"1min":0.69999999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.843,"5min":2.843,"15min":2.843},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.01,"15min":1.01},"min":{"1min":0.80300000000000005,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.385}]},{"osd":7,"last update":"Mon Sep 16 13:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.2490000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509482,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303428,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068412,"statfs":{"total":96636764160,"available":96326053888,"internally_reserved":0,"allocated":1949696,"data_stored":927051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.911,"5min":3.911,"15min":3.911},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.46600000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9729999999999999,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":0.96099999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9510000000000001,"5min":2.9510000000000001,"15min":2.9510000000000001},"last":0.88}]},{"osd":3,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.75800000000000001,"15min":0.72799999999999998},"min":{"1min":0.58199999999999996,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.2089999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":0.80500000000000005,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.76500000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.86899999999999999,"15min":0.84399999999999997},"min":{"1min":0.56299999999999994,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.014},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.96799999999999997,"15min":0.96699999999999997},"min":{"1min":0.51500000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.343}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":0.97499999999999998,"15min":0.94099999999999995},"min":{"1min":0.86499999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":5.0919999999999996,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":1.101},{"interface":"front","average":{"1min":1.163,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.61899999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":3.8279999999999998},"last":0.94999999999999996}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.0429999999999999,"15min":1.0089999999999999},"min":{"1min":0.75700000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.04,"15min":1.0269999999999999},"min":{"1min":0.58299999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":4.0460000000000003,"5min":4.0460000000000003,"15min":4.0460000000000003},"last":1.085}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.995},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.288}]}]},{"osd":7,"up_from":107,"seq":459561500686,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305528,"kb_used_data":1960,"kb_used_omap":50,"kb_used_meta":303501,"kb_avail":94066312,"statfs":{"total":96636764160,"available":96323903488,"internally_reserved":0,"allocated":2007040,"data_stored":1517381,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":51722,"internal_metadata":310785526},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299264,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072576,"statfs":{"total":96636764160,"available":96330317824,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.77800000000000002,"15min":0.76600000000000001},"min":{"1min":0.57399999999999995,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2929999999999999,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96999999999999997,"15min":0.94799999999999995},"min":{"1min":0.52300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.335,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":1.016}]},{"osd":1,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.78500000000000003,"15min":0.77300000000000002},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86899999999999999,"15min":0.86499999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.81899999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84599999999999997,"15min":0.86599999999999999},"min":{"1min":0.60799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.1579999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.044,"5min":0.95999999999999996,"15min":0.95899999999999996},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3979999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.88200000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.97599999999999998,"15min":0.95799999999999996},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.107}]},{"osd":4,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.871,"15min":0.88700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1220000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.874,"5min":1.002,"15min":1.0229999999999999},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3009999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.2509999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.96599999999999997,"15min":0.97799999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.329,"5min":1.637,"15min":1.637},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0529999999999999,"15min":1.0720000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.1910000000000001,"5min":1.661,"15min":1.661},"last":1.1950000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.171},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.224}]}]},{"osd":2,"up_from":14,"seq":60129542185,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307580,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064260,"statfs":{"total":96636764160,"available":96321802240,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59699999999999998,"15min":0.60299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85599999999999998,"5min":1.159,"15min":1.159},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.38600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.506,"15min":1.506},"last":0.33400000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.438,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.05,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.218,"5min":3.298,"15min":3.298},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79600000000000004,"15min":0.80700000000000005},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0980000000000001,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.91800000000000004},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.266,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.97999999999999998,"15min":1.014},"min":{"1min":0.61499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.5780000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.05,"5min":1.002,"15min":1},"min":{"1min":0.57299999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.6279999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.92600000000000005}]},{"osd":5,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0569999999999999,"15min":1.081},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.726,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.131,"5min":1.0880000000000001,"15min":1.1060000000000001},"min":{"1min":0.69799999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5029999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.131}]},{"osd":6,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.054,"15min":1.069},"min":{"1min":0.67600000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.758,"5min":1.857,"15min":1.857},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.141,"15min":1.123},"min":{"1min":0.64600000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4079999999999999,"5min":7.0090000000000003,"15min":7.0090000000000003},"last":1.1779999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.056},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":0.63800000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303464,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068376,"statfs":{"total":96636764160,"available":96326017024,"internally_reserved":0,"allocated":1990656,"data_stored":1517381,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.60699999999999998,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.501,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.01,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.70099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.224,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.53400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.109,"5min":1.74,"15min":1.74},"last":0.91100000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.76200000000000001,"15min":0.77600000000000002},"min":{"1min":0.50900000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.48099999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.276,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1}]},{"osd":4,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.92700000000000005,"15min":0.92500000000000004},"min":{"1min":0.73399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.615,"5min":1.871,"15min":1.871},"last":1.048},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.71499999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.2809999999999999,"5min":1.903,"15min":1.903},"last":0.86199999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0269999999999999,"15min":1.0069999999999999},"min":{"1min":0.80100000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.357,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.242},{"interface":"front","average":{"1min":1.073,"5min":0.97999999999999998,"15min":0.96299999999999997},"min":{"1min":0.70099999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.103}]},{"osd":6,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.044,"15min":1.0409999999999999},"min":{"1min":0.71299999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3879999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0369999999999999,"15min":1.0269999999999999},"min":{"1min":0.75600000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5149999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.0740000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.881,"5min":1.881,"15min":1.881},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0}]}} 2024-09-16T13:19:45.219 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:19:45.511 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:45.513 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:19:45.527 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-16T13:19:45.492851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":214,"num_read":143467,"num_read_kb":680580,"num_write":19270,"num_write_kb":297325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":45250401,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16546,"ondisk_log_size":16546,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2454472,"kb_used_data":15296,"kb_used_omap":62,"kb_used_meta":2438977,"kb_avail":752520248,"statfs":{"total":773094113280,"available":770580733952,"internally_reserved":0,"allocated":15663104,"data_stored":8596916,"data_compressed":67468,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":63650,"internal_metadata":2497513310},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002731"},"pg_stats":[{"pgid":"3.1f","version":"143'370","reported_seq":1980,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473500+0000","last_change":"2024-09-16T13:19:19.135941+0000","last_active":"2024-09-16T13:19:38.473500+0000","last_peered":"2024-09-16T13:19:38.473500+0000","last_clean":"2024-09-16T13:19:38.473500+0000","last_became_active":"2024-09-16T13:18:03.119735+0000","last_became_peered":"2024-09-16T13:18:03.119735+0000","last_unstale":"2024-09-16T13:19:38.473500+0000","last_undegraded":"2024-09-16T13:19:38.473500+0000","last_fullsized":"2024-09-16T13:19:38.473500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:02.061478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012744049999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2248,"num_read_kb":16063,"num_write":361,"num_write_kb":5586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":922127,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"141'824","reported_seq":2700,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336528+0000","last_change":"2024-09-16T13:19:19.131668+0000","last_active":"2024-09-16T13:19:38.336528+0000","last_peered":"2024-09-16T13:19:38.336528+0000","last_clean":"2024-09-16T13:19:38.336528+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:19:38.336528+0000","last_undegraded":"2024-09-16T13:19:38.336528+0000","last_fullsized":"2024-09-16T13:19:38.336528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:50.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003379118,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":5839,"num_read_kb":7327,"num_write":1002,"num_write_kb":6247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"141'446","reported_seq":2670,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473459+0000","last_change":"2024-09-16T13:19:19.136172+0000","last_active":"2024-09-16T13:19:38.473459+0000","last_peered":"2024-09-16T13:19:38.473459+0000","last_clean":"2024-09-16T13:19:38.473459+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:19:38.473459+0000","last_undegraded":"2024-09-16T13:19:38.473459+0000","last_fullsized":"2024-09-16T13:19:38.473459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:14.387827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014771560000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4778,"num_read_kb":13874,"num_write":452,"num_write_kb":6042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"143'449","reported_seq":2727,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.344857+0000","last_change":"2024-09-16T13:19:38.344285+0000","last_active":"2024-09-16T13:19:38.344857+0000","last_peered":"2024-09-16T13:19:38.344857+0000","last_clean":"2024-09-16T13:19:38.344857+0000","last_became_active":"2024-09-16T13:19:38.342560+0000","last_became_peered":"2024-09-16T13:19:38.342560+0000","last_unstale":"2024-09-16T13:19:38.344857+0000","last_undegraded":"2024-09-16T13:19:38.344857+0000","last_fullsized":"2024-09-16T13:19:38.344857+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:57:32.494600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00301937,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":2995,"num_read_kb":25210,"num_write":556,"num_write_kb":12813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"141'671","reported_seq":5617,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343768+0000","last_change":"2024-09-16T13:19:19.164598+0000","last_active":"2024-09-16T13:19:38.343768+0000","last_peered":"2024-09-16T13:19:38.343768+0000","last_clean":"2024-09-16T13:19:38.343768+0000","last_became_active":"2024-09-16T13:19:10.071308+0000","last_became_peered":"2024-09-16T13:19:10.071308+0000","last_unstale":"2024-09-16T13:19:38.343768+0000","last_undegraded":"2024-09-16T13:19:38.343768+0000","last_fullsized":"2024-09-16T13:19:38.343768+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:10:31.562928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002223587,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6572,"num_read_kb":28113,"num_write":920,"num_write_kb":17343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":3647407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"142'661","reported_seq":7962,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338599+0000","last_change":"2024-09-16T13:19:19.129707+0000","last_active":"2024-09-16T13:19:38.338599+0000","last_peered":"2024-09-16T13:19:38.338599+0000","last_clean":"2024-09-16T13:19:38.338599+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:19:38.338599+0000","last_undegraded":"2024-09-16T13:19:38.338599+0000","last_fullsized":"2024-09-16T13:19:38.338599+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:31.071219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028974579999999999,"stat_sum":{"num_bytes":19,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":9256,"num_read_kb":55053,"num_write":857,"num_write_kb":18796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"142'608","reported_seq":5503,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.335998+0000","last_change":"2024-09-16T13:19:19.128105+0000","last_active":"2024-09-16T13:19:38.335998+0000","last_peered":"2024-09-16T13:19:38.335998+0000","last_clean":"2024-09-16T13:19:38.335998+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:19:38.335998+0000","last_undegraded":"2024-09-16T13:19:38.335998+0000","last_fullsized":"2024-09-16T13:19:38.335998+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:42:30.327300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002453879,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":7637,"num_read_kb":32040,"num_write":812,"num_write_kb":13589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'421","reported_seq":2587,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336098+0000","last_change":"2024-09-16T13:19:19.131005+0000","last_active":"2024-09-16T13:19:38.336098+0000","last_peered":"2024-09-16T13:19:38.336098+0000","last_clean":"2024-09-16T13:19:38.336098+0000","last_became_active":"2024-09-16T13:19:10.071112+0000","last_became_peered":"2024-09-16T13:19:10.071112+0000","last_unstale":"2024-09-16T13:19:38.336098+0000","last_undegraded":"2024-09-16T13:19:38.336098+0000","last_fullsized":"2024-09-16T13:19:38.336098+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:29.996047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032828369999999998,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2803,"num_read_kb":8703,"num_write":538,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":228967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"141'573","reported_seq":2387,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343524+0000","last_change":"2024-09-16T13:19:19.163188+0000","last_active":"2024-09-16T13:19:38.343524+0000","last_peered":"2024-09-16T13:19:38.343524+0000","last_clean":"2024-09-16T13:19:38.343524+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:19:38.343524+0000","last_undegraded":"2024-09-16T13:19:38.343524+0000","last_fullsized":"2024-09-16T13:19:38.343524+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:08:57.225261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000869449,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3700,"num_read_kb":13824,"num_write":600,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'410","reported_seq":3104,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473297+0000","last_change":"2024-09-16T13:19:19.135414+0000","last_active":"2024-09-16T13:19:38.473297+0000","last_peered":"2024-09-16T13:19:38.473297+0000","last_clean":"2024-09-16T13:19:38.473297+0000","last_became_active":"2024-09-16T13:18:44.441740+0000","last_became_peered":"2024-09-16T13:18:44.441740+0000","last_unstale":"2024-09-16T13:19:38.473297+0000","last_undegraded":"2024-09-16T13:19:38.473297+0000","last_fullsized":"2024-09-16T13:19:38.473297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:20.994894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082856900000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":5473,"num_read_kb":21309,"num_write":357,"num_write_kb":5655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1185011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"143'229","reported_seq":831,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773785+0000","last_change":"2024-09-16T13:19:19.174836+0000","last_active":"2024-09-16T13:19:38.773785+0000","last_peered":"2024-09-16T13:19:38.773785+0000","last_clean":"2024-09-16T13:19:38.773785+0000","last_became_active":"2024-09-16T13:18:44.435407+0000","last_became_peered":"2024-09-16T13:18:44.435407+0000","last_unstale":"2024-09-16T13:19:38.773785+0000","last_undegraded":"2024-09-16T13:19:38.773785+0000","last_fullsized":"2024-09-16T13:19:38.773785+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:11.546854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016773129999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3249,"num_read_kb":15295,"num_write":190,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1086312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"142'304","reported_seq":5066,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338671+0000","last_change":"2024-09-16T13:19:19.128958+0000","last_active":"2024-09-16T13:19:38.338671+0000","last_peered":"2024-09-16T13:19:38.338671+0000","last_clean":"2024-09-16T13:19:38.338671+0000","last_became_active":"2024-09-16T13:18:47.462532+0000","last_became_peered":"2024-09-16T13:18:47.462532+0000","last_unstale":"2024-09-16T13:19:38.338671+0000","last_undegraded":"2024-09-16T13:19:38.338671+0000","last_fullsized":"2024-09-16T13:19:38.338671+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:16.599784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002111417,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4736,"num_read_kb":25385,"num_write":347,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1053570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.777150+0000","last_change":"2024-09-16T13:19:38.776183+0000","last_active":"2024-09-16T13:19:38.777150+0000","last_peered":"2024-09-16T13:19:38.777150+0000","last_clean":"2024-09-16T13:19:38.777150+0000","last_became_active":"2024-09-16T13:19:38.775280+0000","last_became_peered":"2024-09-16T13:19:38.775280+0000","last_unstale":"2024-09-16T13:19:38.777150+0000","last_undegraded":"2024-09-16T13:19:38.777150+0000","last_fullsized":"2024-09-16T13:19:38.777150+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:46.699452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"143'692","reported_seq":5654,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343415+0000","last_change":"2024-09-16T13:19:19.164842+0000","last_active":"2024-09-16T13:19:38.343415+0000","last_peered":"2024-09-16T13:19:38.343415+0000","last_clean":"2024-09-16T13:19:38.343415+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:19:38.343415+0000","last_undegraded":"2024-09-16T13:19:38.343415+0000","last_fullsized":"2024-09-16T13:19:38.343415+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:52:05.144800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022584039999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6654,"num_read_kb":24625,"num_write":939,"num_write_kb":11031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336150+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:19:38.336150+0000","last_peered":"2024-09-16T13:19:38.336150+0000","last_clean":"2024-09-16T13:19:38.336150+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:19:38.336150+0000","last_undegraded":"2024-09-16T13:19:38.336150+0000","last_fullsized":"2024-09-16T13:19:38.336150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"142'303","reported_seq":1472,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336544+0000","last_change":"2024-09-16T13:19:19.132140+0000","last_active":"2024-09-16T13:19:38.336544+0000","last_peered":"2024-09-16T13:19:38.336544+0000","last_clean":"2024-09-16T13:19:38.336544+0000","last_became_active":"2024-09-16T13:18:47.463039+0000","last_became_peered":"2024-09-16T13:18:47.463039+0000","last_unstale":"2024-09-16T13:19:38.336544+0000","last_undegraded":"2024-09-16T13:19:38.336544+0000","last_fullsized":"2024-09-16T13:19:38.336544+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:05:53.068034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026327310000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":1655,"num_read_kb":8730,"num_write":270,"num_write_kb":5747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1061466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":136,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773757+0000","last_change":"2024-09-16T13:18:44.435471+0000","last_active":"2024-09-16T13:19:38.773757+0000","last_peered":"2024-09-16T13:19:38.773757+0000","last_clean":"2024-09-16T13:19:38.773757+0000","last_became_active":"2024-09-16T13:18:44.434851+0000","last_became_peered":"2024-09-16T13:18:44.434851+0000","last_unstale":"2024-09-16T13:19:38.773757+0000","last_undegraded":"2024-09-16T13:19:38.773757+0000","last_fullsized":"2024-09-16T13:19:38.773757+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:12.980758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"141'372","reported_seq":4687,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336707+0000","last_change":"2024-09-16T13:19:19.129830+0000","last_active":"2024-09-16T13:19:38.336707+0000","last_peered":"2024-09-16T13:19:38.336707+0000","last_clean":"2024-09-16T13:19:38.336707+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:19:38.336707+0000","last_undegraded":"2024-09-16T13:19:38.336707+0000","last_fullsized":"2024-09-16T13:19:38.336707+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:30.996264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032315740000000001,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4693,"num_read_kb":26876,"num_write":422,"num_write_kb":8111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":172,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336904+0000","last_change":"2024-09-16T13:17:42.960492+0000","last_active":"2024-09-16T13:19:38.336904+0000","last_peered":"2024-09-16T13:19:38.336904+0000","last_clean":"2024-09-16T13:19:38.336904+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:19:38.336904+0000","last_undegraded":"2024-09-16T13:19:38.336904+0000","last_fullsized":"2024-09-16T13:19:38.336904+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:35:45.152557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"143'544","reported_seq":3842,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473442+0000","last_change":"2024-09-16T13:19:19.136757+0000","last_active":"2024-09-16T13:19:38.473442+0000","last_peered":"2024-09-16T13:19:38.473442+0000","last_clean":"2024-09-16T13:19:38.473442+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:19:38.473442+0000","last_undegraded":"2024-09-16T13:19:38.473442+0000","last_fullsized":"2024-09-16T13:19:38.473442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:34.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789768,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5947,"num_read_kb":14133,"num_write":543,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"143'506","reported_seq":2497,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.339019+0000","last_change":"2024-09-16T13:19:19.130409+0000","last_active":"2024-09-16T13:19:38.339019+0000","last_peered":"2024-09-16T13:19:38.339019+0000","last_clean":"2024-09-16T13:19:38.339019+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:19:38.339019+0000","last_undegraded":"2024-09-16T13:19:38.339019+0000","last_fullsized":"2024-09-16T13:19:38.339019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:44:16.402879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003357956,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2667,"num_read_kb":21275,"num_write":487,"num_write_kb":9028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":156,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473170+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:19:38.473170+0000","last_peered":"2024-09-16T13:19:38.473170+0000","last_clean":"2024-09-16T13:19:38.473170+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:19:38.473170+0000","last_undegraded":"2024-09-16T13:19:38.473170+0000","last_fullsized":"2024-09-16T13:19:38.473170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"142'424","reported_seq":1787,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.335712+0000","last_change":"2024-09-16T13:19:19.127890+0000","last_active":"2024-09-16T13:19:38.335712+0000","last_peered":"2024-09-16T13:19:38.335712+0000","last_clean":"2024-09-16T13:19:38.335712+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:19:38.335712+0000","last_undegraded":"2024-09-16T13:19:38.335712+0000","last_fullsized":"2024-09-16T13:19:38.335712+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:29.090161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020916900000000002,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3126,"num_read_kb":11030,"num_write":475,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773196+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:19:38.773196+0000","last_peered":"2024-09-16T13:19:38.773196+0000","last_clean":"2024-09-16T13:19:38.773196+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:19:38.773196+0000","last_undegraded":"2024-09-16T13:19:38.773196+0000","last_fullsized":"2024-09-16T13:19:38.773196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'709","reported_seq":2213,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343225+0000","last_change":"2024-09-16T13:19:19.164008+0000","last_active":"2024-09-16T13:19:38.343225+0000","last_peered":"2024-09-16T13:19:38.343225+0000","last_clean":"2024-09-16T13:19:38.343225+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:19:38.343225+0000","last_undegraded":"2024-09-16T13:19:38.343225+0000","last_fullsized":"2024-09-16T13:19:38.343225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:03.357310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017855869999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3881,"num_read_kb":31185,"num_write":963,"num_write_kb":16374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773578+0000","last_change":"2024-09-16T13:18:44.435672+0000","last_active":"2024-09-16T13:19:38.773578+0000","last_peered":"2024-09-16T13:19:38.773578+0000","last_clean":"2024-09-16T13:19:38.773578+0000","last_became_active":"2024-09-16T13:18:44.435188+0000","last_became_peered":"2024-09-16T13:18:44.435188+0000","last_unstale":"2024-09-16T13:19:38.773578+0000","last_undegraded":"2024-09-16T13:19:38.773578+0000","last_fullsized":"2024-09-16T13:19:38.773578+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:35.400164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"143'487","reported_seq":2228,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336304+0000","last_change":"2024-09-16T13:19:19.131189+0000","last_active":"2024-09-16T13:19:38.336304+0000","last_peered":"2024-09-16T13:19:38.336304+0000","last_clean":"2024-09-16T13:19:38.336304+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:19:38.336304+0000","last_undegraded":"2024-09-16T13:19:38.336304+0000","last_fullsized":"2024-09-16T13:19:38.336304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:45.154984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003557705,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3512,"num_read_kb":15515,"num_write":504,"num_write_kb":6261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473138+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:19:38.473138+0000","last_peered":"2024-09-16T13:19:38.473138+0000","last_clean":"2024-09-16T13:19:38.473138+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:19:38.473138+0000","last_undegraded":"2024-09-16T13:19:38.473138+0000","last_fullsized":"2024-09-16T13:19:38.473138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":236,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773593+0000","last_change":"2024-09-16T13:18:44.437802+0000","last_active":"2024-09-16T13:19:38.773593+0000","last_peered":"2024-09-16T13:19:38.773593+0000","last_clean":"2024-09-16T13:19:38.773593+0000","last_became_active":"2024-09-16T13:18:44.437418+0000","last_became_peered":"2024-09-16T13:18:44.437418+0000","last_unstale":"2024-09-16T13:19:38.773593+0000","last_undegraded":"2024-09-16T13:19:38.773593+0000","last_fullsized":"2024-09-16T13:19:38.773593+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:29.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"143'606","reported_seq":1309,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336209+0000","last_change":"2024-09-16T13:19:19.131515+0000","last_active":"2024-09-16T13:19:38.336209+0000","last_peered":"2024-09-16T13:19:38.336209+0000","last_clean":"2024-09-16T13:19:38.336209+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:19:38.336209+0000","last_undegraded":"2024-09-16T13:19:38.336209+0000","last_fullsized":"2024-09-16T13:19:38.336209+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:59.515115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033814320000000002,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2017,"num_read_kb":19372,"num_write":789,"num_write_kb":12171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"142'523","reported_seq":3009,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338761+0000","last_change":"2024-09-16T13:19:19.128362+0000","last_active":"2024-09-16T13:19:38.338761+0000","last_peered":"2024-09-16T13:19:38.338761+0000","last_clean":"2024-09-16T13:19:38.338761+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:19:38.338761+0000","last_undegraded":"2024-09-16T13:19:38.338761+0000","last_fullsized":"2024-09-16T13:19:38.338761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:49:10.301550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001533006,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4183,"num_read_kb":31351,"num_write":445,"num_write_kb":12351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"142'942","reported_seq":3022,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.347467+0000","last_change":"2024-09-16T13:19:38.346887+0000","last_active":"2024-09-16T13:19:38.347467+0000","last_peered":"2024-09-16T13:19:38.347467+0000","last_clean":"2024-09-16T13:19:38.347467+0000","last_became_active":"2024-09-16T13:19:38.343200+0000","last_became_peered":"2024-09-16T13:19:38.343200+0000","last_unstale":"2024-09-16T13:19:38.347467+0000","last_undegraded":"2024-09-16T13:19:38.347467+0000","last_fullsized":"2024-09-16T13:19:38.347467+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:37.001790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025018559999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":4754,"num_read_kb":19969,"num_write":1275,"num_write_kb":12323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"142'1140","reported_seq":4459,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336696+0000","last_change":"2024-09-16T13:19:19.128947+0000","last_active":"2024-09-16T13:19:38.336696+0000","last_peered":"2024-09-16T13:19:38.336696+0000","last_clean":"2024-09-16T13:19:38.336696+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:19:38.336696+0000","last_undegraded":"2024-09-16T13:19:38.336696+0000","last_fullsized":"2024-09-16T13:19:38.336696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:20.684083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014023239999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":9870,"num_read_kb":27372,"num_write":1699,"num_write_kb":10648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"143'323","reported_seq":1457,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473160+0000","last_change":"2024-09-16T13:19:19.135617+0000","last_active":"2024-09-16T13:19:38.473160+0000","last_peered":"2024-09-16T13:19:38.473160+0000","last_clean":"2024-09-16T13:19:38.473160+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:19:38.473160+0000","last_undegraded":"2024-09-16T13:19:38.473160+0000","last_fullsized":"2024-09-16T13:19:38.473160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:39.937083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010610159999999999,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":1447,"num_read_kb":15039,"num_write":293,"num_write_kb":7034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'389","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774008+0000","last_change":"2024-09-16T13:19:19.175390+0000","last_active":"2024-09-16T13:19:38.774008+0000","last_peered":"2024-09-16T13:19:38.774008+0000","last_clean":"2024-09-16T13:19:38.774008+0000","last_became_active":"2024-09-16T13:18:47.468001+0000","last_became_peered":"2024-09-16T13:18:47.468001+0000","last_unstale":"2024-09-16T13:19:38.774008+0000","last_undegraded":"2024-09-16T13:19:38.774008+0000","last_fullsized":"2024-09-16T13:19:38.774008+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:11:32.886561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018957069999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2747,"num_read_kb":18459,"num_write":343,"num_write_kb":14196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"142'510","reported_seq":7096,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773857+0000","last_change":"2024-09-16T13:19:19.173823+0000","last_active":"2024-09-16T13:19:38.773857+0000","last_peered":"2024-09-16T13:19:38.773857+0000","last_clean":"2024-09-16T13:19:38.773857+0000","last_became_active":"2024-09-16T13:18:47.467028+0000","last_became_peered":"2024-09-16T13:18:47.467028+0000","last_unstale":"2024-09-16T13:19:38.773857+0000","last_undegraded":"2024-09-16T13:19:38.773857+0000","last_fullsized":"2024-09-16T13:19:38.773857+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:35.852378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092761899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":8104,"num_read_kb":29095,"num_write":570,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1715710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"142'310","reported_seq":1005,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774295+0000","last_change":"2024-09-16T13:19:19.173006+0000","last_active":"2024-09-16T13:19:38.774295+0000","last_peered":"2024-09-16T13:19:38.774295+0000","last_clean":"2024-09-16T13:19:38.774295+0000","last_became_active":"2024-09-16T13:18:44.444025+0000","last_became_peered":"2024-09-16T13:18:44.444025+0000","last_unstale":"2024-09-16T13:19:38.774295+0000","last_undegraded":"2024-09-16T13:19:38.774295+0000","last_fullsized":"2024-09-16T13:19:38.774295+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:25:03.521529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056513400000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":692,"num_read_kb":10388,"num_write":284,"num_write_kb":7715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":63719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"142'265","reported_seq":1394,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.346281+0000","last_change":"2024-09-16T13:19:38.345793+0000","last_active":"2024-09-16T13:19:38.346281+0000","last_peered":"2024-09-16T13:19:38.346281+0000","last_clean":"2024-09-16T13:19:38.346281+0000","last_became_active":"2024-09-16T13:19:38.344229+0000","last_became_peered":"2024-09-16T13:19:38.344229+0000","last_unstale":"2024-09-16T13:19:38.346281+0000","last_undegraded":"2024-09-16T13:19:38.346281+0000","last_fullsized":"2024-09-16T13:19:38.346281+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:30:56.026666+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00077869099999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":784,"num_read_kb":16305,"num_write":257,"num_write_kb":11082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"142'556","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773502+0000","last_change":"2024-09-16T13:19:19.175226+0000","last_active":"2024-09-16T13:19:38.773502+0000","last_peered":"2024-09-16T13:19:38.773502+0000","last_clean":"2024-09-16T13:19:38.773502+0000","last_became_active":"2024-09-16T13:18:47.462561+0000","last_became_peered":"2024-09-16T13:18:47.462561+0000","last_unstale":"2024-09-16T13:19:38.773502+0000","last_undegraded":"2024-09-16T13:19:38.773502+0000","last_fullsized":"2024-09-16T13:19:38.773502+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.079652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021368229999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3858,"num_read_kb":23843,"num_write":732,"num_write_kb":11756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"141'398","reported_seq":7509,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343225+0000","last_change":"2024-09-16T13:19:19.164016+0000","last_active":"2024-09-16T13:19:38.343225+0000","last_peered":"2024-09-16T13:19:38.343225+0000","last_clean":"2024-09-16T13:19:38.343225+0000","last_became_active":"2024-09-16T13:18:47.462412+0000","last_became_peered":"2024-09-16T13:18:47.462412+0000","last_unstale":"2024-09-16T13:19:38.343225+0000","last_undegraded":"2024-09-16T13:19:38.343225+0000","last_fullsized":"2024-09-16T13:19:38.343225+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:50:25.256657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001704484,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":8501,"num_read_kb":35036,"num_write":362,"num_write_kb":9220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2494648,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"141'506","reported_seq":2090,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773579+0000","last_change":"2024-09-16T13:19:19.172960+0000","last_active":"2024-09-16T13:19:38.773579+0000","last_peered":"2024-09-16T13:19:38.773579+0000","last_clean":"2024-09-16T13:19:38.773579+0000","last_became_active":"2024-09-16T13:18:44.443832+0000","last_became_peered":"2024-09-16T13:18:44.443832+0000","last_unstale":"2024-09-16T13:19:38.773579+0000","last_undegraded":"2024-09-16T13:19:38.773579+0000","last_fullsized":"2024-09-16T13:19:38.773579+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:45.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001469344,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":5023,"num_read_kb":18731,"num_write":523,"num_write_kb":7705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1653183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":218,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":214,"num_read":143401,"num_read_kb":680525,"num_write":19167,"num_write_kb":295695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":44660033,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0},"log_size":16471,"ondisk_log_size":16471,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509483,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311732,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060108,"statfs":{"total":96636764160,"available":96317550592,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.84999999999999998},"min":{"1min":0.47299999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2809999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.499,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1910000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.61299999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.83799999999999997,"15min":0.82799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.3009999999999999,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.98799999999999999,"15min":0.98199999999999998},"min":{"1min":0.64500000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.6789999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.878,"15min":0.89400000000000002},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.268,"5min":1.476,"15min":1.476},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.97699999999999998,"15min":1.0389999999999999},"min":{"1min":0.63300000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.278,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":0.82699999999999996}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.0409999999999999,"15min":1.081},"min":{"1min":0.73099999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.5350000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.98599999999999999,"15min":0.999},"min":{"1min":0.65200000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.256,"5min":3.0209999999999999,"15min":3.0209999999999999},"last":0.71699999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.96599999999999997,"15min":0.98799999999999999},"min":{"1min":0.53800000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.335,"5min":1.825,"15min":1.825},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0309999999999999,"15min":1.0600000000000001},"min":{"1min":0.56299999999999994,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.347,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.056}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88100000000000001,"15min":0.90300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.409,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.1100000000000001,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":1.151}]},{"osd":7,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.98199999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509482,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311740,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060100,"statfs":{"total":96636764160,"available":96317542400,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.78300000000000003,"15min":0.80100000000000005},"min":{"1min":0.51600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.927,"15min":1.927},"last":1.282}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78700000000000003,"15min":0.78300000000000003},"min":{"1min":0.52300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.90000000000000002,"15min":0.91600000000000004},"min":{"1min":0.55000000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.3109999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.82499999999999996,"15min":0.82199999999999995},"min":{"1min":0.59699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.93500000000000005},"min":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0209999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.83599999999999997,"15min":0.84399999999999997},"min":{"1min":0.69699999999999995,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.167},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.94699999999999995,"15min":0.95399999999999996},"min":{"1min":0.85999999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.1339999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.96199999999999997},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.4239999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.341,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94599999999999995,"15min":0.97999999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.0609999999999999,"15min":1.0920000000000001},"min":{"1min":0.59899999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.369}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.2490000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509483,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311736,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060104,"statfs":{"total":96636764160,"available":96317546496,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.45400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65000000000000002,"15min":0.66300000000000003},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92600000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.76600000000000001},"min":{"1min":0.48399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0660000000000001,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.96999999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.76300000000000001,"15min":0.77100000000000002},"min":{"1min":0.57399999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.88500000000000001},"min":{"1min":0.56399999999999995,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1879999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":1.254}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0209999999999999,"15min":0.98899999999999999},"min":{"1min":0.747,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.7989999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":0.754},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.98999999999999999,"15min":0.96499999999999997},"min":{"1min":0.77500000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3100000000000001,"5min":4.9489999999999998,"15min":4.9489999999999998},"last":1.083}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.99299999999999999,"15min":0.95599999999999996},"min":{"1min":0.71499999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3380000000000001,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.087,"15min":1.091},"min":{"1min":0.86499999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.45,"15min":1.45},"last":1.1850000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0649999999999999,"15min":1.0629999999999999},"min":{"1min":0.69999999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.843,"5min":2.843,"15min":2.843},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.01,"15min":1.01},"min":{"1min":0.80300000000000005,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.94099999999999995}]},{"osd":7,"last update":"Mon Sep 16 13:19:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.996,"5min":0.996,"15min":0.996},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.292},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":1.149}]}]},{"osd":1,"up_from":15,"seq":64424509483,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303428,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068412,"statfs":{"total":96636764160,"available":96326053888,"internally_reserved":0,"allocated":1949696,"data_stored":927051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.911,"5min":3.911,"15min":3.911},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.46600000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9729999999999999,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9510000000000001,"5min":2.9510000000000001,"15min":2.9510000000000001},"last":0.90500000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.75800000000000001,"15min":0.72799999999999998},"min":{"1min":0.58199999999999996,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.2089999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":0.80500000000000005,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.83999999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.86899999999999999,"15min":0.84399999999999997},"min":{"1min":0.56299999999999994,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.96799999999999997,"15min":0.96699999999999997},"min":{"1min":0.51500000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.03}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":0.97499999999999998,"15min":0.94099999999999995},"min":{"1min":0.86499999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":5.0919999999999996,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.163,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.61899999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":3.8279999999999998},"last":1.0009999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.0429999999999999,"15min":1.0089999999999999},"min":{"1min":0.75700000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.04,"15min":1.0269999999999999},"min":{"1min":0.58299999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":4.0460000000000003,"5min":4.0460000000000003,"15min":4.0460000000000003},"last":1.1719999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.1539999999999999}]}]},{"osd":7,"up_from":107,"seq":459561500686,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305528,"kb_used_data":1960,"kb_used_omap":50,"kb_used_meta":303501,"kb_avail":94066312,"statfs":{"total":96636764160,"available":96323903488,"internally_reserved":0,"allocated":2007040,"data_stored":1517381,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":51722,"internal_metadata":310785526},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299264,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072576,"statfs":{"total":96636764160,"available":96330317824,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.77800000000000002,"15min":0.76600000000000001},"min":{"1min":0.57399999999999995,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2929999999999999,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96999999999999997,"15min":0.94799999999999995},"min":{"1min":0.52300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.335,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":1.016}]},{"osd":1,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.78500000000000003,"15min":0.77300000000000002},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86899999999999999,"15min":0.86499999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.81899999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84599999999999997,"15min":0.86599999999999999},"min":{"1min":0.60799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.1579999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.044,"5min":0.95999999999999996,"15min":0.95899999999999996},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3979999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.88200000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.97599999999999998,"15min":0.95799999999999996},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.107}]},{"osd":4,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.871,"15min":0.88700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1220000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.874,"5min":1.002,"15min":1.0229999999999999},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3009999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.2509999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.96599999999999997,"15min":0.97799999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.329,"5min":1.637,"15min":1.637},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0529999999999999,"15min":1.0720000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.1910000000000001,"5min":1.661,"15min":1.661},"last":1.1950000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.171},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.224}]}]},{"osd":2,"up_from":14,"seq":60129542185,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307580,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064260,"statfs":{"total":96636764160,"available":96321802240,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59699999999999998,"15min":0.60299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85599999999999998,"5min":1.159,"15min":1.159},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.38600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.506,"15min":1.506},"last":0.33400000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.438,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.05,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.218,"5min":3.298,"15min":3.298},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79600000000000004,"15min":0.80700000000000005},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0980000000000001,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.91800000000000004},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.266,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.97999999999999998,"15min":1.014},"min":{"1min":0.61499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.5780000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.05,"5min":1.002,"15min":1},"min":{"1min":0.57299999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.6279999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.92600000000000005}]},{"osd":5,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0569999999999999,"15min":1.081},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.726,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.131,"5min":1.0880000000000001,"15min":1.1060000000000001},"min":{"1min":0.69799999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5029999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.131}]},{"osd":6,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.054,"15min":1.069},"min":{"1min":0.67600000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.758,"5min":1.857,"15min":1.857},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.141,"15min":1.123},"min":{"1min":0.64600000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4079999999999999,"5min":7.0090000000000003,"15min":7.0090000000000003},"last":1.1779999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.056},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":0.63800000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303464,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068376,"statfs":{"total":96636764160,"available":96326017024,"internally_reserved":0,"allocated":1990656,"data_stored":1517381,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.60699999999999998,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.501,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.01,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.70099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.224,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.53400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.109,"5min":1.74,"15min":1.74},"last":0.91100000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.76200000000000001,"15min":0.77600000000000002},"min":{"1min":0.50900000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.48099999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.276,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1}]},{"osd":4,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.92700000000000005,"15min":0.92500000000000004},"min":{"1min":0.73399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.615,"5min":1.871,"15min":1.871},"last":1.048},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.71499999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.2809999999999999,"5min":1.903,"15min":1.903},"last":0.86199999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0269999999999999,"15min":1.0069999999999999},"min":{"1min":0.80100000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.357,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.242},{"interface":"front","average":{"1min":1.073,"5min":0.97999999999999998,"15min":0.96299999999999997},"min":{"1min":0.70099999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.103}]},{"osd":6,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.044,"15min":1.0409999999999999},"min":{"1min":0.71299999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3879999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0369999999999999,"15min":1.0269999999999999},"min":{"1min":0.75600000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5149999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.0740000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.881,"5min":1.881,"15min":1.881},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0}]}} 2024-09-16T13:19:45.530 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T13:19:45.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:19:45.826 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:45.828 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:19:45.840 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-16T13:19:45.492851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":214,"num_read":143467,"num_read_kb":680580,"num_write":19270,"num_write_kb":297325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":45250401,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16546,"ondisk_log_size":16546,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2454472,"kb_used_data":15296,"kb_used_omap":62,"kb_used_meta":2438977,"kb_avail":752520248,"statfs":{"total":773094113280,"available":770580733952,"internally_reserved":0,"allocated":15663104,"data_stored":8596916,"data_compressed":67468,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":63650,"internal_metadata":2497513310},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002731"},"pg_stats":[{"pgid":"3.1f","version":"143'370","reported_seq":1980,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473500+0000","last_change":"2024-09-16T13:19:19.135941+0000","last_active":"2024-09-16T13:19:38.473500+0000","last_peered":"2024-09-16T13:19:38.473500+0000","last_clean":"2024-09-16T13:19:38.473500+0000","last_became_active":"2024-09-16T13:18:03.119735+0000","last_became_peered":"2024-09-16T13:18:03.119735+0000","last_unstale":"2024-09-16T13:19:38.473500+0000","last_undegraded":"2024-09-16T13:19:38.473500+0000","last_fullsized":"2024-09-16T13:19:38.473500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:02.061478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012744049999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2248,"num_read_kb":16063,"num_write":361,"num_write_kb":5586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":922127,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"141'824","reported_seq":2700,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336528+0000","last_change":"2024-09-16T13:19:19.131668+0000","last_active":"2024-09-16T13:19:38.336528+0000","last_peered":"2024-09-16T13:19:38.336528+0000","last_clean":"2024-09-16T13:19:38.336528+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:19:38.336528+0000","last_undegraded":"2024-09-16T13:19:38.336528+0000","last_fullsized":"2024-09-16T13:19:38.336528+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:50.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003379118,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":5839,"num_read_kb":7327,"num_write":1002,"num_write_kb":6247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"141'446","reported_seq":2670,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473459+0000","last_change":"2024-09-16T13:19:19.136172+0000","last_active":"2024-09-16T13:19:38.473459+0000","last_peered":"2024-09-16T13:19:38.473459+0000","last_clean":"2024-09-16T13:19:38.473459+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:19:38.473459+0000","last_undegraded":"2024-09-16T13:19:38.473459+0000","last_fullsized":"2024-09-16T13:19:38.473459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:14.387827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014771560000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4778,"num_read_kb":13874,"num_write":452,"num_write_kb":6042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"143'449","reported_seq":2727,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.344857+0000","last_change":"2024-09-16T13:19:38.344285+0000","last_active":"2024-09-16T13:19:38.344857+0000","last_peered":"2024-09-16T13:19:38.344857+0000","last_clean":"2024-09-16T13:19:38.344857+0000","last_became_active":"2024-09-16T13:19:38.342560+0000","last_became_peered":"2024-09-16T13:19:38.342560+0000","last_unstale":"2024-09-16T13:19:38.344857+0000","last_undegraded":"2024-09-16T13:19:38.344857+0000","last_fullsized":"2024-09-16T13:19:38.344857+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:57:32.494600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00301937,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":2995,"num_read_kb":25210,"num_write":556,"num_write_kb":12813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"141'671","reported_seq":5617,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343768+0000","last_change":"2024-09-16T13:19:19.164598+0000","last_active":"2024-09-16T13:19:38.343768+0000","last_peered":"2024-09-16T13:19:38.343768+0000","last_clean":"2024-09-16T13:19:38.343768+0000","last_became_active":"2024-09-16T13:19:10.071308+0000","last_became_peered":"2024-09-16T13:19:10.071308+0000","last_unstale":"2024-09-16T13:19:38.343768+0000","last_undegraded":"2024-09-16T13:19:38.343768+0000","last_fullsized":"2024-09-16T13:19:38.343768+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:10:31.562928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002223587,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6572,"num_read_kb":28113,"num_write":920,"num_write_kb":17343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":3647407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"142'661","reported_seq":7962,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338599+0000","last_change":"2024-09-16T13:19:19.129707+0000","last_active":"2024-09-16T13:19:38.338599+0000","last_peered":"2024-09-16T13:19:38.338599+0000","last_clean":"2024-09-16T13:19:38.338599+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:19:38.338599+0000","last_undegraded":"2024-09-16T13:19:38.338599+0000","last_fullsized":"2024-09-16T13:19:38.338599+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:08:31.071219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028974579999999999,"stat_sum":{"num_bytes":19,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":9256,"num_read_kb":55053,"num_write":857,"num_write_kb":18796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"142'608","reported_seq":5503,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.335998+0000","last_change":"2024-09-16T13:19:19.128105+0000","last_active":"2024-09-16T13:19:38.335998+0000","last_peered":"2024-09-16T13:19:38.335998+0000","last_clean":"2024-09-16T13:19:38.335998+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:19:38.335998+0000","last_undegraded":"2024-09-16T13:19:38.335998+0000","last_fullsized":"2024-09-16T13:19:38.335998+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:42:30.327300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002453879,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":7637,"num_read_kb":32040,"num_write":812,"num_write_kb":13589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'421","reported_seq":2587,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336098+0000","last_change":"2024-09-16T13:19:19.131005+0000","last_active":"2024-09-16T13:19:38.336098+0000","last_peered":"2024-09-16T13:19:38.336098+0000","last_clean":"2024-09-16T13:19:38.336098+0000","last_became_active":"2024-09-16T13:19:10.071112+0000","last_became_peered":"2024-09-16T13:19:10.071112+0000","last_unstale":"2024-09-16T13:19:38.336098+0000","last_undegraded":"2024-09-16T13:19:38.336098+0000","last_fullsized":"2024-09-16T13:19:38.336098+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:34:29.996047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032828369999999998,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2803,"num_read_kb":8703,"num_write":538,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":228967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"141'573","reported_seq":2387,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343524+0000","last_change":"2024-09-16T13:19:19.163188+0000","last_active":"2024-09-16T13:19:38.343524+0000","last_peered":"2024-09-16T13:19:38.343524+0000","last_clean":"2024-09-16T13:19:38.343524+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:19:38.343524+0000","last_undegraded":"2024-09-16T13:19:38.343524+0000","last_fullsized":"2024-09-16T13:19:38.343524+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:08:57.225261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000869449,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3700,"num_read_kb":13824,"num_write":600,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'410","reported_seq":3104,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473297+0000","last_change":"2024-09-16T13:19:19.135414+0000","last_active":"2024-09-16T13:19:38.473297+0000","last_peered":"2024-09-16T13:19:38.473297+0000","last_clean":"2024-09-16T13:19:38.473297+0000","last_became_active":"2024-09-16T13:18:44.441740+0000","last_became_peered":"2024-09-16T13:18:44.441740+0000","last_unstale":"2024-09-16T13:19:38.473297+0000","last_undegraded":"2024-09-16T13:19:38.473297+0000","last_fullsized":"2024-09-16T13:19:38.473297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:20.994894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082856900000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":5473,"num_read_kb":21309,"num_write":357,"num_write_kb":5655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1185011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"143'229","reported_seq":831,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773785+0000","last_change":"2024-09-16T13:19:19.174836+0000","last_active":"2024-09-16T13:19:38.773785+0000","last_peered":"2024-09-16T13:19:38.773785+0000","last_clean":"2024-09-16T13:19:38.773785+0000","last_became_active":"2024-09-16T13:18:44.435407+0000","last_became_peered":"2024-09-16T13:18:44.435407+0000","last_unstale":"2024-09-16T13:19:38.773785+0000","last_undegraded":"2024-09-16T13:19:38.773785+0000","last_fullsized":"2024-09-16T13:19:38.773785+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:11.546854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016773129999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3249,"num_read_kb":15295,"num_write":190,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1086312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"142'304","reported_seq":5066,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338671+0000","last_change":"2024-09-16T13:19:19.128958+0000","last_active":"2024-09-16T13:19:38.338671+0000","last_peered":"2024-09-16T13:19:38.338671+0000","last_clean":"2024-09-16T13:19:38.338671+0000","last_became_active":"2024-09-16T13:18:47.462532+0000","last_became_peered":"2024-09-16T13:18:47.462532+0000","last_unstale":"2024-09-16T13:19:38.338671+0000","last_undegraded":"2024-09-16T13:19:38.338671+0000","last_fullsized":"2024-09-16T13:19:38.338671+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:16.599784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002111417,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4736,"num_read_kb":25385,"num_write":347,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1053570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.777150+0000","last_change":"2024-09-16T13:19:38.776183+0000","last_active":"2024-09-16T13:19:38.777150+0000","last_peered":"2024-09-16T13:19:38.777150+0000","last_clean":"2024-09-16T13:19:38.777150+0000","last_became_active":"2024-09-16T13:19:38.775280+0000","last_became_peered":"2024-09-16T13:19:38.775280+0000","last_unstale":"2024-09-16T13:19:38.777150+0000","last_undegraded":"2024-09-16T13:19:38.777150+0000","last_fullsized":"2024-09-16T13:19:38.777150+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:46.699452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"143'692","reported_seq":5654,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343415+0000","last_change":"2024-09-16T13:19:19.164842+0000","last_active":"2024-09-16T13:19:38.343415+0000","last_peered":"2024-09-16T13:19:38.343415+0000","last_clean":"2024-09-16T13:19:38.343415+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:19:38.343415+0000","last_undegraded":"2024-09-16T13:19:38.343415+0000","last_fullsized":"2024-09-16T13:19:38.343415+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:52:05.144800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022584039999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6654,"num_read_kb":24625,"num_write":939,"num_write_kb":11031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336150+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:19:38.336150+0000","last_peered":"2024-09-16T13:19:38.336150+0000","last_clean":"2024-09-16T13:19:38.336150+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:19:38.336150+0000","last_undegraded":"2024-09-16T13:19:38.336150+0000","last_fullsized":"2024-09-16T13:19:38.336150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:38.368192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"142'303","reported_seq":1472,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336544+0000","last_change":"2024-09-16T13:19:19.132140+0000","last_active":"2024-09-16T13:19:38.336544+0000","last_peered":"2024-09-16T13:19:38.336544+0000","last_clean":"2024-09-16T13:19:38.336544+0000","last_became_active":"2024-09-16T13:18:47.463039+0000","last_became_peered":"2024-09-16T13:18:47.463039+0000","last_unstale":"2024-09-16T13:19:38.336544+0000","last_undegraded":"2024-09-16T13:19:38.336544+0000","last_fullsized":"2024-09-16T13:19:38.336544+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:05:53.068034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026327310000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":1655,"num_read_kb":8730,"num_write":270,"num_write_kb":5747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1061466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":136,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773757+0000","last_change":"2024-09-16T13:18:44.435471+0000","last_active":"2024-09-16T13:19:38.773757+0000","last_peered":"2024-09-16T13:19:38.773757+0000","last_clean":"2024-09-16T13:19:38.773757+0000","last_became_active":"2024-09-16T13:18:44.434851+0000","last_became_peered":"2024-09-16T13:18:44.434851+0000","last_unstale":"2024-09-16T13:19:38.773757+0000","last_undegraded":"2024-09-16T13:19:38.773757+0000","last_fullsized":"2024-09-16T13:19:38.773757+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:12.980758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"141'372","reported_seq":4687,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336707+0000","last_change":"2024-09-16T13:19:19.129830+0000","last_active":"2024-09-16T13:19:38.336707+0000","last_peered":"2024-09-16T13:19:38.336707+0000","last_clean":"2024-09-16T13:19:38.336707+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:19:38.336707+0000","last_undegraded":"2024-09-16T13:19:38.336707+0000","last_fullsized":"2024-09-16T13:19:38.336707+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:30.996264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032315740000000001,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4693,"num_read_kb":26876,"num_write":422,"num_write_kb":8111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":172,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336904+0000","last_change":"2024-09-16T13:17:42.960492+0000","last_active":"2024-09-16T13:19:38.336904+0000","last_peered":"2024-09-16T13:19:38.336904+0000","last_clean":"2024-09-16T13:19:38.336904+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:19:38.336904+0000","last_undegraded":"2024-09-16T13:19:38.336904+0000","last_fullsized":"2024-09-16T13:19:38.336904+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:35:45.152557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"143'544","reported_seq":3842,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473442+0000","last_change":"2024-09-16T13:19:19.136757+0000","last_active":"2024-09-16T13:19:38.473442+0000","last_peered":"2024-09-16T13:19:38.473442+0000","last_clean":"2024-09-16T13:19:38.473442+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:19:38.473442+0000","last_undegraded":"2024-09-16T13:19:38.473442+0000","last_fullsized":"2024-09-16T13:19:38.473442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:34.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789768,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5947,"num_read_kb":14133,"num_write":543,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"143'506","reported_seq":2497,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.339019+0000","last_change":"2024-09-16T13:19:19.130409+0000","last_active":"2024-09-16T13:19:38.339019+0000","last_peered":"2024-09-16T13:19:38.339019+0000","last_clean":"2024-09-16T13:19:38.339019+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:19:38.339019+0000","last_undegraded":"2024-09-16T13:19:38.339019+0000","last_fullsized":"2024-09-16T13:19:38.339019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:44:16.402879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003357956,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2667,"num_read_kb":21275,"num_write":487,"num_write_kb":9028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":156,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473170+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:19:38.473170+0000","last_peered":"2024-09-16T13:19:38.473170+0000","last_clean":"2024-09-16T13:19:38.473170+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:19:38.473170+0000","last_undegraded":"2024-09-16T13:19:38.473170+0000","last_fullsized":"2024-09-16T13:19:38.473170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"142'424","reported_seq":1787,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.335712+0000","last_change":"2024-09-16T13:19:19.127890+0000","last_active":"2024-09-16T13:19:38.335712+0000","last_peered":"2024-09-16T13:19:38.335712+0000","last_clean":"2024-09-16T13:19:38.335712+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:19:38.335712+0000","last_undegraded":"2024-09-16T13:19:38.335712+0000","last_fullsized":"2024-09-16T13:19:38.335712+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:51:29.090161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020916900000000002,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3126,"num_read_kb":11030,"num_write":475,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773196+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:19:38.773196+0000","last_peered":"2024-09-16T13:19:38.773196+0000","last_clean":"2024-09-16T13:19:38.773196+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:19:38.773196+0000","last_undegraded":"2024-09-16T13:19:38.773196+0000","last_fullsized":"2024-09-16T13:19:38.773196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'709","reported_seq":2213,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343225+0000","last_change":"2024-09-16T13:19:19.164008+0000","last_active":"2024-09-16T13:19:38.343225+0000","last_peered":"2024-09-16T13:19:38.343225+0000","last_clean":"2024-09-16T13:19:38.343225+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:19:38.343225+0000","last_undegraded":"2024-09-16T13:19:38.343225+0000","last_fullsized":"2024-09-16T13:19:38.343225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:03.357310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017855869999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3881,"num_read_kb":31185,"num_write":963,"num_write_kb":16374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773578+0000","last_change":"2024-09-16T13:18:44.435672+0000","last_active":"2024-09-16T13:19:38.773578+0000","last_peered":"2024-09-16T13:19:38.773578+0000","last_clean":"2024-09-16T13:19:38.773578+0000","last_became_active":"2024-09-16T13:18:44.435188+0000","last_became_peered":"2024-09-16T13:18:44.435188+0000","last_unstale":"2024-09-16T13:19:38.773578+0000","last_undegraded":"2024-09-16T13:19:38.773578+0000","last_fullsized":"2024-09-16T13:19:38.773578+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:35.400164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"143'487","reported_seq":2228,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336304+0000","last_change":"2024-09-16T13:19:19.131189+0000","last_active":"2024-09-16T13:19:38.336304+0000","last_peered":"2024-09-16T13:19:38.336304+0000","last_clean":"2024-09-16T13:19:38.336304+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:19:38.336304+0000","last_undegraded":"2024-09-16T13:19:38.336304+0000","last_fullsized":"2024-09-16T13:19:38.336304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:41:45.154984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003557705,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3512,"num_read_kb":15515,"num_write":504,"num_write_kb":6261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473138+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:19:38.473138+0000","last_peered":"2024-09-16T13:19:38.473138+0000","last_clean":"2024-09-16T13:19:38.473138+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:19:38.473138+0000","last_undegraded":"2024-09-16T13:19:38.473138+0000","last_fullsized":"2024-09-16T13:19:38.473138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":236,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773593+0000","last_change":"2024-09-16T13:18:44.437802+0000","last_active":"2024-09-16T13:19:38.773593+0000","last_peered":"2024-09-16T13:19:38.773593+0000","last_clean":"2024-09-16T13:19:38.773593+0000","last_became_active":"2024-09-16T13:18:44.437418+0000","last_became_peered":"2024-09-16T13:18:44.437418+0000","last_unstale":"2024-09-16T13:19:38.773593+0000","last_undegraded":"2024-09-16T13:19:38.773593+0000","last_fullsized":"2024-09-16T13:19:38.773593+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:29.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"143'606","reported_seq":1309,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336209+0000","last_change":"2024-09-16T13:19:19.131515+0000","last_active":"2024-09-16T13:19:38.336209+0000","last_peered":"2024-09-16T13:19:38.336209+0000","last_clean":"2024-09-16T13:19:38.336209+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:19:38.336209+0000","last_undegraded":"2024-09-16T13:19:38.336209+0000","last_fullsized":"2024-09-16T13:19:38.336209+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:23:59.515115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033814320000000002,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2017,"num_read_kb":19372,"num_write":789,"num_write_kb":12171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"142'523","reported_seq":3009,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.338761+0000","last_change":"2024-09-16T13:19:19.128362+0000","last_active":"2024-09-16T13:19:38.338761+0000","last_peered":"2024-09-16T13:19:38.338761+0000","last_clean":"2024-09-16T13:19:38.338761+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:19:38.338761+0000","last_undegraded":"2024-09-16T13:19:38.338761+0000","last_fullsized":"2024-09-16T13:19:38.338761+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:49:10.301550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001533006,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4183,"num_read_kb":31351,"num_write":445,"num_write_kb":12351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"142'942","reported_seq":3022,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.347467+0000","last_change":"2024-09-16T13:19:38.346887+0000","last_active":"2024-09-16T13:19:38.347467+0000","last_peered":"2024-09-16T13:19:38.347467+0000","last_clean":"2024-09-16T13:19:38.347467+0000","last_became_active":"2024-09-16T13:19:38.343200+0000","last_became_peered":"2024-09-16T13:19:38.343200+0000","last_unstale":"2024-09-16T13:19:38.347467+0000","last_undegraded":"2024-09-16T13:19:38.347467+0000","last_fullsized":"2024-09-16T13:19:38.347467+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:37.001790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025018559999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":4754,"num_read_kb":19969,"num_write":1275,"num_write_kb":12323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"142'1140","reported_seq":4459,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.336696+0000","last_change":"2024-09-16T13:19:19.128947+0000","last_active":"2024-09-16T13:19:38.336696+0000","last_peered":"2024-09-16T13:19:38.336696+0000","last_clean":"2024-09-16T13:19:38.336696+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:19:38.336696+0000","last_undegraded":"2024-09-16T13:19:38.336696+0000","last_fullsized":"2024-09-16T13:19:38.336696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:50:20.684083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014023239999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":9870,"num_read_kb":27372,"num_write":1699,"num_write_kb":10648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"143'323","reported_seq":1457,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473160+0000","last_change":"2024-09-16T13:19:19.135617+0000","last_active":"2024-09-16T13:19:38.473160+0000","last_peered":"2024-09-16T13:19:38.473160+0000","last_clean":"2024-09-16T13:19:38.473160+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:19:38.473160+0000","last_undegraded":"2024-09-16T13:19:38.473160+0000","last_fullsized":"2024-09-16T13:19:38.473160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:39.937083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010610159999999999,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":1447,"num_read_kb":15039,"num_write":293,"num_write_kb":7034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'389","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774008+0000","last_change":"2024-09-16T13:19:19.175390+0000","last_active":"2024-09-16T13:19:38.774008+0000","last_peered":"2024-09-16T13:19:38.774008+0000","last_clean":"2024-09-16T13:19:38.774008+0000","last_became_active":"2024-09-16T13:18:47.468001+0000","last_became_peered":"2024-09-16T13:18:47.468001+0000","last_unstale":"2024-09-16T13:19:38.774008+0000","last_undegraded":"2024-09-16T13:19:38.774008+0000","last_fullsized":"2024-09-16T13:19:38.774008+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:11:32.886561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018957069999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2747,"num_read_kb":18459,"num_write":343,"num_write_kb":14196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"142'510","reported_seq":7096,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773857+0000","last_change":"2024-09-16T13:19:19.173823+0000","last_active":"2024-09-16T13:19:38.773857+0000","last_peered":"2024-09-16T13:19:38.773857+0000","last_clean":"2024-09-16T13:19:38.773857+0000","last_became_active":"2024-09-16T13:18:47.467028+0000","last_became_peered":"2024-09-16T13:18:47.467028+0000","last_unstale":"2024-09-16T13:19:38.773857+0000","last_undegraded":"2024-09-16T13:19:38.773857+0000","last_fullsized":"2024-09-16T13:19:38.773857+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:35.852378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092761899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":8104,"num_read_kb":29095,"num_write":570,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1715710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"142'310","reported_seq":1005,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774295+0000","last_change":"2024-09-16T13:19:19.173006+0000","last_active":"2024-09-16T13:19:38.774295+0000","last_peered":"2024-09-16T13:19:38.774295+0000","last_clean":"2024-09-16T13:19:38.774295+0000","last_became_active":"2024-09-16T13:18:44.444025+0000","last_became_peered":"2024-09-16T13:18:44.444025+0000","last_unstale":"2024-09-16T13:19:38.774295+0000","last_undegraded":"2024-09-16T13:19:38.774295+0000","last_fullsized":"2024-09-16T13:19:38.774295+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:25:03.521529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056513400000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":692,"num_read_kb":10388,"num_write":284,"num_write_kb":7715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":63719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"142'265","reported_seq":1394,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.346281+0000","last_change":"2024-09-16T13:19:38.345793+0000","last_active":"2024-09-16T13:19:38.346281+0000","last_peered":"2024-09-16T13:19:38.346281+0000","last_clean":"2024-09-16T13:19:38.346281+0000","last_became_active":"2024-09-16T13:19:38.344229+0000","last_became_peered":"2024-09-16T13:19:38.344229+0000","last_unstale":"2024-09-16T13:19:38.346281+0000","last_undegraded":"2024-09-16T13:19:38.346281+0000","last_fullsized":"2024-09-16T13:19:38.346281+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:30:56.026666+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00077869099999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":784,"num_read_kb":16305,"num_write":257,"num_write_kb":11082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"142'556","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773502+0000","last_change":"2024-09-16T13:19:19.175226+0000","last_active":"2024-09-16T13:19:38.773502+0000","last_peered":"2024-09-16T13:19:38.773502+0000","last_clean":"2024-09-16T13:19:38.773502+0000","last_became_active":"2024-09-16T13:18:47.462561+0000","last_became_peered":"2024-09-16T13:18:47.462561+0000","last_unstale":"2024-09-16T13:19:38.773502+0000","last_undegraded":"2024-09-16T13:19:38.773502+0000","last_fullsized":"2024-09-16T13:19:38.773502+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.079652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021368229999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3858,"num_read_kb":23843,"num_write":732,"num_write_kb":11756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"141'398","reported_seq":7509,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.343225+0000","last_change":"2024-09-16T13:19:19.164016+0000","last_active":"2024-09-16T13:19:38.343225+0000","last_peered":"2024-09-16T13:19:38.343225+0000","last_clean":"2024-09-16T13:19:38.343225+0000","last_became_active":"2024-09-16T13:18:47.462412+0000","last_became_peered":"2024-09-16T13:18:47.462412+0000","last_unstale":"2024-09-16T13:19:38.343225+0000","last_undegraded":"2024-09-16T13:19:38.343225+0000","last_fullsized":"2024-09-16T13:19:38.343225+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:50:25.256657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001704484,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":8501,"num_read_kb":35036,"num_write":362,"num_write_kb":9220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2494648,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"141'506","reported_seq":2090,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773579+0000","last_change":"2024-09-16T13:19:19.172960+0000","last_active":"2024-09-16T13:19:38.773579+0000","last_peered":"2024-09-16T13:19:38.773579+0000","last_clean":"2024-09-16T13:19:38.773579+0000","last_became_active":"2024-09-16T13:18:44.443832+0000","last_became_peered":"2024-09-16T13:18:44.443832+0000","last_unstale":"2024-09-16T13:19:38.773579+0000","last_undegraded":"2024-09-16T13:19:38.773579+0000","last_fullsized":"2024-09-16T13:19:38.773579+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:45.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001469344,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":5023,"num_read_kb":18731,"num_write":523,"num_write_kb":7705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1653183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":218,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":214,"num_read":143401,"num_read_kb":680525,"num_write":19167,"num_write_kb":295695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":44660033,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0},"log_size":16471,"ondisk_log_size":16471,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509483,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311732,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060108,"statfs":{"total":96636764160,"available":96317550592,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.84999999999999998},"min":{"1min":0.47299999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2809999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.499,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1910000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.61299999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.83799999999999997,"15min":0.82799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.3009999999999999,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.98799999999999999,"15min":0.98199999999999998},"min":{"1min":0.64500000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.6789999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.878,"15min":0.89400000000000002},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.268,"5min":1.476,"15min":1.476},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.97699999999999998,"15min":1.0389999999999999},"min":{"1min":0.63300000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.278,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":0.82699999999999996}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.0409999999999999,"15min":1.081},"min":{"1min":0.73099999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.5350000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.98599999999999999,"15min":0.999},"min":{"1min":0.65200000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.256,"5min":3.0209999999999999,"15min":3.0209999999999999},"last":0.71699999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.96599999999999997,"15min":0.98799999999999999},"min":{"1min":0.53800000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.335,"5min":1.825,"15min":1.825},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0309999999999999,"15min":1.0600000000000001},"min":{"1min":0.56299999999999994,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.347,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.056}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88100000000000001,"15min":0.90300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.409,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.1100000000000001,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":1.151}]},{"osd":7,"last update":"Mon Sep 16 13:17:35 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.139,"15min":1.139},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.98199999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509482,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311740,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060100,"statfs":{"total":96636764160,"available":96317542400,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.78300000000000003,"15min":0.80100000000000005},"min":{"1min":0.51600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.927,"15min":1.927},"last":1.282}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78700000000000003,"15min":0.78300000000000003},"min":{"1min":0.52300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.90000000000000002,"15min":0.91600000000000004},"min":{"1min":0.55000000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.3109999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.82499999999999996,"15min":0.82199999999999995},"min":{"1min":0.59699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.93500000000000005},"min":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0209999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.83599999999999997,"15min":0.84399999999999997},"min":{"1min":0.69699999999999995,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.167},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.94699999999999995,"15min":0.95399999999999996},"min":{"1min":0.85999999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.1339999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.96199999999999997},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.4239999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.341,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.0620000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94599999999999995,"15min":0.97999999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.0609999999999999,"15min":1.0920000000000001},"min":{"1min":0.59899999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.369}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.694,"5min":1.694,"15min":1.694},"last":1.2490000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509483,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311736,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060104,"statfs":{"total":96636764160,"available":96317546496,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.45400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65000000000000002,"15min":0.66300000000000003},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92600000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.76600000000000001},"min":{"1min":0.48399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0660000000000001,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.96999999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.76300000000000001,"15min":0.77100000000000002},"min":{"1min":0.57399999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.88500000000000001},"min":{"1min":0.56399999999999995,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1879999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":1.254}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0209999999999999,"15min":0.98899999999999999},"min":{"1min":0.747,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.7989999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":0.754},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.98999999999999999,"15min":0.96499999999999997},"min":{"1min":0.77500000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3100000000000001,"5min":4.9489999999999998,"15min":4.9489999999999998},"last":1.083}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.99299999999999999,"15min":0.95599999999999996},"min":{"1min":0.71499999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3380000000000001,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.087,"15min":1.091},"min":{"1min":0.86499999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.45,"15min":1.45},"last":1.1850000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0649999999999999,"15min":1.0629999999999999},"min":{"1min":0.69999999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.843,"5min":2.843,"15min":2.843},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.01,"15min":1.01},"min":{"1min":0.80300000000000005,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.94099999999999995}]},{"osd":7,"last update":"Mon Sep 16 13:19:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.996,"5min":0.996,"15min":0.996},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.292},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":1.149}]}]},{"osd":1,"up_from":15,"seq":64424509483,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303428,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068412,"statfs":{"total":96636764160,"available":96326053888,"internally_reserved":0,"allocated":1949696,"data_stored":927051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.911,"5min":3.911,"15min":3.911},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.46600000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9729999999999999,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9510000000000001,"5min":2.9510000000000001,"15min":2.9510000000000001},"last":0.90500000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.75800000000000001,"15min":0.72799999999999998},"min":{"1min":0.58199999999999996,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.2089999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.1599999999999999,"5min":0.80500000000000005,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.83999999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.86899999999999999,"15min":0.84399999999999997},"min":{"1min":0.56299999999999994,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.96799999999999997,"15min":0.96699999999999997},"min":{"1min":0.51500000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.03}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":0.97499999999999998,"15min":0.94099999999999995},"min":{"1min":0.86499999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":5.0919999999999996,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.163,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.61899999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":3.8279999999999998},"last":1.0009999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.0429999999999999,"15min":1.0089999999999999},"min":{"1min":0.75700000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.04,"15min":1.0269999999999999},"min":{"1min":0.58299999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":4.0460000000000003,"5min":4.0460000000000003,"15min":4.0460000000000003},"last":1.1719999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.1539999999999999}]}]},{"osd":7,"up_from":107,"seq":459561500686,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305528,"kb_used_data":1960,"kb_used_omap":50,"kb_used_meta":303501,"kb_avail":94066312,"statfs":{"total":96636764160,"available":96323903488,"internally_reserved":0,"allocated":2007040,"data_stored":1517381,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":51722,"internal_metadata":310785526},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299264,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072576,"statfs":{"total":96636764160,"available":96330317824,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.77800000000000002,"15min":0.76600000000000001},"min":{"1min":0.57399999999999995,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2929999999999999,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96999999999999997,"15min":0.94799999999999995},"min":{"1min":0.52300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.335,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":1.016}]},{"osd":1,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.78500000000000003,"15min":0.77300000000000002},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.86899999999999999,"15min":0.86499999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.81899999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84599999999999997,"15min":0.86599999999999999},"min":{"1min":0.60799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.1579999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.044,"5min":0.95999999999999996,"15min":0.95899999999999996},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3979999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.88200000000000001}]},{"osd":3,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.97599999999999998,"15min":0.95799999999999996},"min":{"1min":0.503,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.107}]},{"osd":4,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.871,"15min":0.88700000000000001},"min":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1220000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.874,"5min":1.002,"15min":1.0229999999999999},"min":{"1min":0.42299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3009999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.2509999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:18:42 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.96599999999999997,"15min":0.97799999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.329,"5min":1.637,"15min":1.637},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0529999999999999,"15min":1.0720000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.1910000000000001,"5min":1.661,"15min":1.661},"last":1.1950000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.171},{"interface":"front","average":{"1min":1.153,"5min":1.153,"15min":1.153},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.224}]}]},{"osd":2,"up_from":14,"seq":60129542185,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307580,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064260,"statfs":{"total":96636764160,"available":96321802240,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59699999999999998,"15min":0.60299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85599999999999998,"5min":1.159,"15min":1.159},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.38600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.506,"15min":1.506},"last":0.33400000000000002}]},{"osd":1,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.438,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.05,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.218,"5min":3.298,"15min":3.298},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79600000000000004,"15min":0.80700000000000005},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0980000000000001,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.91800000000000004},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.266,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.97999999999999998,"15min":1.014},"min":{"1min":0.61499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.5780000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.05,"5min":1.002,"15min":1},"min":{"1min":0.57299999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.6279999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.92600000000000005}]},{"osd":5,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0569999999999999,"15min":1.081},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.726,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.131,"5min":1.0880000000000001,"15min":1.1060000000000001},"min":{"1min":0.69799999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5029999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.131}]},{"osd":6,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.054,"15min":1.069},"min":{"1min":0.67600000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.758,"5min":1.857,"15min":1.857},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.141,"15min":1.123},"min":{"1min":0.64600000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4079999999999999,"5min":7.0090000000000003,"15min":7.0090000000000003},"last":1.1779999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.056},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.419,"5min":2.419,"15min":2.419},"last":0.63800000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303464,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068376,"statfs":{"total":96636764160,"available":96326017024,"internally_reserved":0,"allocated":1990656,"data_stored":1517381,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.60699999999999998,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.501,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.01,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.70099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.224,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.53400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.109,"5min":1.74,"15min":1.74},"last":0.91100000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.76200000000000001,"15min":0.77600000000000002},"min":{"1min":0.50900000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.48099999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.276,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1}]},{"osd":4,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.92700000000000005,"15min":0.92500000000000004},"min":{"1min":0.73399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.615,"5min":1.871,"15min":1.871},"last":1.048},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.71499999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.2809999999999999,"5min":1.903,"15min":1.903},"last":0.86199999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0269999999999999,"15min":1.0069999999999999},"min":{"1min":0.80100000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.357,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.242},{"interface":"front","average":{"1min":1.073,"5min":0.97999999999999998,"15min":0.96299999999999997},"min":{"1min":0.70099999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.103}]},{"osd":6,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.044,"15min":1.0409999999999999},"min":{"1min":0.71299999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3879999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0369999999999999,"15min":1.0269999999999999},"min":{"1min":0.75600000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5149999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.0740000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:17:33 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.881,"5min":1.881,"15min":1.881},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.518,"5min":1.518,"15min":1.518},"last":1.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0}]}} 2024-09-16T13:19:45.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T13:19:46.134 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:46.134 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":162,"fsid":"6ddcae89-6c18-4eb7-8712-1c53f3b07278","created":"2024-09-16T13:16:23.176689+0000","modified":"2024-09-16T13:19:39.331549+0000","last_up_change":"2024-09-16T13:18:43.413345+0000","last_in_change":"2024-09-16T13:17:42.548353+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T13:16:33.497561+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T13:16:40.994538+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":21,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-16T13:16:50.759705+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":81,"snap_epoch":143,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"be8543b4-3d28-4b37-a4ce-d2a56a09aeaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":22271},{"type":"v1","addr":"172.21.15.98:6809","nonce":22271}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":22271},{"type":"v1","addr":"172.21.15.98:6811","nonce":22271}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":22271},{"type":"v1","addr":"172.21.15.98:6817","nonce":22271}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6813","nonce":22271},{"type":"v1","addr":"172.21.15.98:6814","nonce":22271}]},"public_addr":"172.21.15.98:6809/22271","cluster_addr":"172.21.15.98:6811/22271","heartbeat_back_addr":"172.21.15.98:6817/22271","heartbeat_front_addr":"172.21.15.98:6814/22271","state":["exists","up"]},{"osd":1,"uuid":"a7f95c7b-56d4-437b-a160-cabb8fd2b1e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":22272},{"type":"v1","addr":"172.21.15.98:6815","nonce":22272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":22272},{"type":"v1","addr":"172.21.15.98:6819","nonce":22272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":22272},{"type":"v1","addr":"172.21.15.98:6823","nonce":22272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":22272},{"type":"v1","addr":"172.21.15.98:6821","nonce":22272}]},"public_addr":"172.21.15.98:6815/22272","cluster_addr":"172.21.15.98:6819/22272","heartbeat_back_addr":"172.21.15.98:6823/22272","heartbeat_front_addr":"172.21.15.98:6821/22272","state":["exists","up"]},{"osd":2,"uuid":"a9c92ac4-c55f-43e5-a194-9ffb3f63ef3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":22273},{"type":"v1","addr":"172.21.15.98:6801","nonce":22273}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":22273},{"type":"v1","addr":"172.21.15.98:6803","nonce":22273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":22273},{"type":"v1","addr":"172.21.15.98:6807","nonce":22273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":22273},{"type":"v1","addr":"172.21.15.98:6805","nonce":22273}]},"public_addr":"172.21.15.98:6801/22273","cluster_addr":"172.21.15.98:6803/22273","heartbeat_back_addr":"172.21.15.98:6807/22273","heartbeat_front_addr":"172.21.15.98:6805/22273","state":["exists","up"]},{"osd":3,"uuid":"4898b5b5-b912-430c-adcb-c52d90226917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":22270},{"type":"v1","addr":"172.21.15.98:6825","nonce":22270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":22270},{"type":"v1","addr":"172.21.15.98:6827","nonce":22270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":22270},{"type":"v1","addr":"172.21.15.98:6831","nonce":22270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":22270},{"type":"v1","addr":"172.21.15.98:6829","nonce":22270}]},"public_addr":"172.21.15.98:6825/22270","cluster_addr":"172.21.15.98:6827/22270","heartbeat_back_addr":"172.21.15.98:6831/22270","heartbeat_front_addr":"172.21.15.98:6829/22270","state":["exists","up"]},{"osd":4,"uuid":"6c478be2-f606-41d8-967c-9a689f09a440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":22286},{"type":"v1","addr":"172.21.15.151:6825","nonce":22286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":22286},{"type":"v1","addr":"172.21.15.151:6827","nonce":22286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":22286},{"type":"v1","addr":"172.21.15.151:6831","nonce":22286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":22286},{"type":"v1","addr":"172.21.15.151:6829","nonce":22286}]},"public_addr":"172.21.15.151:6825/22286","cluster_addr":"172.21.15.151:6827/22286","heartbeat_back_addr":"172.21.15.151:6831/22286","heartbeat_front_addr":"172.21.15.151:6829/22286","state":["exists","up"]},{"osd":5,"uuid":"5001c828-b223-4d2c-b893-f9c56da73b83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":22285},{"type":"v1","addr":"172.21.15.151:6809","nonce":22285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":22285},{"type":"v1","addr":"172.21.15.151:6811","nonce":22285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":22285},{"type":"v1","addr":"172.21.15.151:6815","nonce":22285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":22285},{"type":"v1","addr":"172.21.15.151:6813","nonce":22285}]},"public_addr":"172.21.15.151:6809/22285","cluster_addr":"172.21.15.151:6811/22285","heartbeat_back_addr":"172.21.15.151:6815/22285","heartbeat_front_addr":"172.21.15.151:6813/22285","state":["exists","up"]},{"osd":6,"uuid":"6f7a9125-674c-4be2-93e1-b6296cd00f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":22315},{"type":"v1","addr":"172.21.15.151:6817","nonce":22315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":22315},{"type":"v1","addr":"172.21.15.151:6819","nonce":22315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":22315},{"type":"v1","addr":"172.21.15.151:6823","nonce":22315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":22315},{"type":"v1","addr":"172.21.15.151:6821","nonce":22315}]},"public_addr":"172.21.15.151:6817/22315","cluster_addr":"172.21.15.151:6819/22315","heartbeat_back_addr":"172.21.15.151:6823/22315","heartbeat_front_addr":"172.21.15.151:6821/22315","state":["exists","up"]},{"osd":7,"uuid":"bcdfe435-f24d-45a9-a629-7380ad2e575a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":81,"up_from":107,"up_thru":137,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":33053},{"type":"v1","addr":"172.21.15.151:6801","nonce":33053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":33053},{"type":"v1","addr":"172.21.15.151:6803","nonce":33053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":33053},{"type":"v1","addr":"172.21.15.151:6807","nonce":33053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":33053},{"type":"v1","addr":"172.21.15.151:6805","nonce":33053}]},"public_addr":"172.21.15.151:6801/33053","cluster_addr":"172.21.15.151:6803/33053","heartbeat_back_addr":"172.21.15.151:6807/33053","heartbeat_front_addr":"172.21.15.151:6805/33053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.191311+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.563551+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.493457+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.599815+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.634352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.262873+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.508881+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T13:16:30.348276+0000","dead_epoch":82}],"pg_upmap":[{"pgid":"3.1c","osds":[6,1]},{"pgid":"3.1f","osds":[5,3]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:19:47.149 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T13:19:47.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:47.310 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:47.310 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:47.310 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:47.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T13:19:47.624 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T13:19:47.633 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T13:19:47.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:47.777 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:47.777 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:47.777 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:47.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T13:19:48.078 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T13:19:48.088 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T13:19:48.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:48.234 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:48.234 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:48.234 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:48.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T13:19:48.549 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T13:19:48.559 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T13:19:48.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:48.699 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:48.699 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:48.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T13:19:49.009 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T13:19:49.019 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T13:19:49.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:49.160 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "" 2024-09-16T13:19:49.160 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:49.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T13:19:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T13:19:49.484 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T13:19:49.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:49.628 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:49.628 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:49.629 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:49.641 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T13:19:49.933 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T13:19:49.942 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T13:19:49.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:50.082 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:50.082 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:50.082 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:50.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T13:19:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T13:19:50.399 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T13:19:50.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:19:50.537 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T13:19:50.538 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:19:50.538 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T13:19:50.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T13:19:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T13:19:50.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:19:51.145 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:19:51.148 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:19:51.159 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-16T13:19:49.494056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":214,"num_read":143467,"num_read_kb":680580,"num_write":19270,"num_write_kb":297325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":45250401,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16552,"ondisk_log_size":16552,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2454472,"kb_used_data":15296,"kb_used_omap":62,"kb_used_meta":2438977,"kb_avail":752520248,"statfs":{"total":773094113280,"available":770580733952,"internally_reserved":0,"allocated":15663104,"data_stored":8596916,"data_compressed":67468,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":63650,"internal_metadata":2497513310},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.138788"},"pg_stats":[{"pgid":"3.1f","version":"143'370","reported_seq":1980,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473500+0000","last_change":"2024-09-16T13:19:19.135941+0000","last_active":"2024-09-16T13:19:38.473500+0000","last_peered":"2024-09-16T13:19:38.473500+0000","last_clean":"2024-09-16T13:19:38.473500+0000","last_became_active":"2024-09-16T13:18:03.119735+0000","last_became_peered":"2024-09-16T13:18:03.119735+0000","last_unstale":"2024-09-16T13:19:38.473500+0000","last_undegraded":"2024-09-16T13:19:38.473500+0000","last_fullsized":"2024-09-16T13:19:38.473500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:19:02.061478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012744049999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2248,"num_read_kb":16063,"num_write":361,"num_write_kb":5586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":922127,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"162'830","reported_seq":2717,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:49.351578+0000","last_change":"2024-09-16T13:19:49.351553+0000","last_active":"2024-09-16T13:19:49.351578+0000","last_peered":"2024-09-16T13:19:49.351578+0000","last_clean":"2024-09-16T13:19:49.351578+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:19:49.351578+0000","last_undegraded":"2024-09-16T13:19:49.351578+0000","last_fullsized":"2024-09-16T13:19:49.351578+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'830","last_scrub_stamp":"2024-09-16T13:19:49.351459+0000","last_deep_scrub":"162'830","last_deep_scrub_stamp":"2024-09-16T13:19:49.351459+0000","last_clean_scrub_stamp":"2024-09-16T13:19:49.351459+0000","objects_scrubbed":6,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:35.542928+0000","scrub_duration":0.013408455,"objects_trimmed":0,"snaptrim_duration":0.003379118,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":5839,"num_read_kb":7327,"num_write":1002,"num_write_kb":6247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"141'446","reported_seq":2670,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473459+0000","last_change":"2024-09-16T13:19:19.136172+0000","last_active":"2024-09-16T13:19:38.473459+0000","last_peered":"2024-09-16T13:19:38.473459+0000","last_clean":"2024-09-16T13:19:38.473459+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:19:38.473459+0000","last_undegraded":"2024-09-16T13:19:38.473459+0000","last_fullsized":"2024-09-16T13:19:38.473459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:55:14.387827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014771560000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4778,"num_read_kb":13874,"num_write":452,"num_write_kb":6042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"143'449","reported_seq":2727,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.344857+0000","last_change":"2024-09-16T13:19:38.344285+0000","last_active":"2024-09-16T13:19:38.344857+0000","last_peered":"2024-09-16T13:19:38.344857+0000","last_clean":"2024-09-16T13:19:38.344857+0000","last_became_active":"2024-09-16T13:19:38.342560+0000","last_became_peered":"2024-09-16T13:19:38.342560+0000","last_unstale":"2024-09-16T13:19:38.344857+0000","last_undegraded":"2024-09-16T13:19:38.344857+0000","last_fullsized":"2024-09-16T13:19:38.344857+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:57:32.494600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00301937,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":2995,"num_read_kb":25210,"num_write":556,"num_write_kb":12813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"141'671","reported_seq":5618,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.431987+0000","last_change":"2024-09-16T13:19:19.164598+0000","last_active":"2024-09-16T13:19:39.431987+0000","last_peered":"2024-09-16T13:19:39.431987+0000","last_clean":"2024-09-16T13:19:39.431987+0000","last_became_active":"2024-09-16T13:19:10.071308+0000","last_became_peered":"2024-09-16T13:19:10.071308+0000","last_unstale":"2024-09-16T13:19:39.431987+0000","last_undegraded":"2024-09-16T13:19:39.431987+0000","last_fullsized":"2024-09-16T13:19:39.431987+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002223587,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6572,"num_read_kb":28113,"num_write":920,"num_write_kb":17343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":3647407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"142'661","reported_seq":7963,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.432343+0000","last_change":"2024-09-16T13:19:19.129707+0000","last_active":"2024-09-16T13:19:39.432343+0000","last_peered":"2024-09-16T13:19:39.432343+0000","last_clean":"2024-09-16T13:19:39.432343+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:19:39.432343+0000","last_undegraded":"2024-09-16T13:19:39.432343+0000","last_fullsized":"2024-09-16T13:19:39.432343+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028974579999999999,"stat_sum":{"num_bytes":19,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":9256,"num_read_kb":55053,"num_write":857,"num_write_kb":18796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"142'608","reported_seq":5504,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.431565+0000","last_change":"2024-09-16T13:19:19.128105+0000","last_active":"2024-09-16T13:19:39.431565+0000","last_peered":"2024-09-16T13:19:39.431565+0000","last_clean":"2024-09-16T13:19:39.431565+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:19:39.431565+0000","last_undegraded":"2024-09-16T13:19:39.431565+0000","last_fullsized":"2024-09-16T13:19:39.431565+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002453879,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":7637,"num_read_kb":32040,"num_write":812,"num_write_kb":13589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"141'421","reported_seq":2588,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.338933+0000","last_change":"2024-09-16T13:19:19.131005+0000","last_active":"2024-09-16T13:19:39.338933+0000","last_peered":"2024-09-16T13:19:39.338933+0000","last_clean":"2024-09-16T13:19:39.338933+0000","last_became_active":"2024-09-16T13:19:10.071112+0000","last_became_peered":"2024-09-16T13:19:10.071112+0000","last_unstale":"2024-09-16T13:19:39.338933+0000","last_undegraded":"2024-09-16T13:19:39.338933+0000","last_fullsized":"2024-09-16T13:19:39.338933+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032828369999999998,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2803,"num_read_kb":8703,"num_write":538,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":228967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"141'573","reported_seq":2388,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.431938+0000","last_change":"2024-09-16T13:19:19.163188+0000","last_active":"2024-09-16T13:19:39.431938+0000","last_peered":"2024-09-16T13:19:39.431938+0000","last_clean":"2024-09-16T13:19:39.431938+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:19:39.431938+0000","last_undegraded":"2024-09-16T13:19:39.431938+0000","last_fullsized":"2024-09-16T13:19:39.431938+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000869449,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3700,"num_read_kb":13824,"num_write":600,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'410","reported_seq":3104,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473297+0000","last_change":"2024-09-16T13:19:19.135414+0000","last_active":"2024-09-16T13:19:38.473297+0000","last_peered":"2024-09-16T13:19:38.473297+0000","last_clean":"2024-09-16T13:19:38.473297+0000","last_became_active":"2024-09-16T13:18:44.441740+0000","last_became_peered":"2024-09-16T13:18:44.441740+0000","last_unstale":"2024-09-16T13:19:38.473297+0000","last_undegraded":"2024-09-16T13:19:38.473297+0000","last_fullsized":"2024-09-16T13:19:38.473297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:06:20.994894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082856900000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":5473,"num_read_kb":21309,"num_write":357,"num_write_kb":5655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1185011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"143'229","reported_seq":831,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773785+0000","last_change":"2024-09-16T13:19:19.174836+0000","last_active":"2024-09-16T13:19:38.773785+0000","last_peered":"2024-09-16T13:19:38.773785+0000","last_clean":"2024-09-16T13:19:38.773785+0000","last_became_active":"2024-09-16T13:18:44.435407+0000","last_became_peered":"2024-09-16T13:18:44.435407+0000","last_unstale":"2024-09-16T13:19:38.773785+0000","last_undegraded":"2024-09-16T13:19:38.773785+0000","last_fullsized":"2024-09-16T13:19:38.773785+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:09:11.546854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016773129999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3249,"num_read_kb":15295,"num_write":190,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1086312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"142'304","reported_seq":5067,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.432612+0000","last_change":"2024-09-16T13:19:19.128958+0000","last_active":"2024-09-16T13:19:39.432612+0000","last_peered":"2024-09-16T13:19:39.432612+0000","last_clean":"2024-09-16T13:19:39.432612+0000","last_became_active":"2024-09-16T13:18:47.462532+0000","last_became_peered":"2024-09-16T13:18:47.462532+0000","last_unstale":"2024-09-16T13:19:39.432612+0000","last_undegraded":"2024-09-16T13:19:39.432612+0000","last_fullsized":"2024-09-16T13:19:39.432612+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002111417,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4736,"num_read_kb":25385,"num_write":347,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1053570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.777150+0000","last_change":"2024-09-16T13:19:38.776183+0000","last_active":"2024-09-16T13:19:38.777150+0000","last_peered":"2024-09-16T13:19:38.777150+0000","last_clean":"2024-09-16T13:19:38.777150+0000","last_became_active":"2024-09-16T13:19:38.775280+0000","last_became_peered":"2024-09-16T13:19:38.775280+0000","last_unstale":"2024-09-16T13:19:38.777150+0000","last_undegraded":"2024-09-16T13:19:38.777150+0000","last_fullsized":"2024-09-16T13:19:38.777150+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:46.699452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"143'692","reported_seq":5655,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.432005+0000","last_change":"2024-09-16T13:19:19.164842+0000","last_active":"2024-09-16T13:19:39.432005+0000","last_peered":"2024-09-16T13:19:39.432005+0000","last_clean":"2024-09-16T13:19:39.432005+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:19:39.432005+0000","last_undegraded":"2024-09-16T13:19:39.432005+0000","last_fullsized":"2024-09-16T13:19:39.432005+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022584039999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6654,"num_read_kb":24625,"num_write":939,"num_write_kb":11031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":155,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.431620+0000","last_change":"2024-09-16T13:16:44.845595+0000","last_active":"2024-09-16T13:19:39.431620+0000","last_peered":"2024-09-16T13:19:39.431620+0000","last_clean":"2024-09-16T13:19:39.431620+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:19:39.431620+0000","last_undegraded":"2024-09-16T13:19:39.431620+0000","last_fullsized":"2024-09-16T13:19:39.431620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"142'303","reported_seq":1473,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.339399+0000","last_change":"2024-09-16T13:19:19.132140+0000","last_active":"2024-09-16T13:19:39.339399+0000","last_peered":"2024-09-16T13:19:39.339399+0000","last_clean":"2024-09-16T13:19:39.339399+0000","last_became_active":"2024-09-16T13:18:47.463039+0000","last_became_peered":"2024-09-16T13:18:47.463039+0000","last_unstale":"2024-09-16T13:19:39.339399+0000","last_undegraded":"2024-09-16T13:19:39.339399+0000","last_fullsized":"2024-09-16T13:19:39.339399+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026327310000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":1655,"num_read_kb":8730,"num_write":270,"num_write_kb":5747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1061466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":136,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773757+0000","last_change":"2024-09-16T13:18:44.435471+0000","last_active":"2024-09-16T13:19:38.773757+0000","last_peered":"2024-09-16T13:19:38.773757+0000","last_clean":"2024-09-16T13:19:38.773757+0000","last_became_active":"2024-09-16T13:18:44.434851+0000","last_became_peered":"2024-09-16T13:18:44.434851+0000","last_unstale":"2024-09-16T13:19:38.773757+0000","last_undegraded":"2024-09-16T13:19:38.773757+0000","last_fullsized":"2024-09-16T13:19:38.773757+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:56:12.980758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"141'372","reported_seq":4688,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.431996+0000","last_change":"2024-09-16T13:19:19.129830+0000","last_active":"2024-09-16T13:19:39.431996+0000","last_peered":"2024-09-16T13:19:39.431996+0000","last_clean":"2024-09-16T13:19:39.431996+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:19:39.431996+0000","last_undegraded":"2024-09-16T13:19:39.431996+0000","last_fullsized":"2024-09-16T13:19:39.431996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032315740000000001,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4693,"num_read_kb":26876,"num_write":422,"num_write_kb":8111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":173,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.432088+0000","last_change":"2024-09-16T13:17:42.960492+0000","last_active":"2024-09-16T13:19:39.432088+0000","last_peered":"2024-09-16T13:19:39.432088+0000","last_clean":"2024-09-16T13:19:39.432088+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:19:39.432088+0000","last_undegraded":"2024-09-16T13:19:39.432088+0000","last_fullsized":"2024-09-16T13:19:39.432088+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"143'544","reported_seq":3842,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473442+0000","last_change":"2024-09-16T13:19:19.136757+0000","last_active":"2024-09-16T13:19:38.473442+0000","last_peered":"2024-09-16T13:19:38.473442+0000","last_clean":"2024-09-16T13:19:38.473442+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:19:38.473442+0000","last_undegraded":"2024-09-16T13:19:38.473442+0000","last_fullsized":"2024-09-16T13:19:38.473442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:27:34.704120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789768,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5947,"num_read_kb":14133,"num_write":543,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"143'506","reported_seq":2498,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.432656+0000","last_change":"2024-09-16T13:19:19.130409+0000","last_active":"2024-09-16T13:19:39.432656+0000","last_peered":"2024-09-16T13:19:39.432656+0000","last_clean":"2024-09-16T13:19:39.432656+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:19:39.432656+0000","last_undegraded":"2024-09-16T13:19:39.432656+0000","last_fullsized":"2024-09-16T13:19:39.432656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003357956,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2667,"num_read_kb":21275,"num_write":487,"num_write_kb":9028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"21'2","reported_seq":156,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473170+0000","last_change":"2024-09-16T13:16:44.843137+0000","last_active":"2024-09-16T13:19:38.473170+0000","last_peered":"2024-09-16T13:19:38.473170+0000","last_clean":"2024-09-16T13:19:38.473170+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:19:38.473170+0000","last_undegraded":"2024-09-16T13:19:38.473170+0000","last_fullsized":"2024-09-16T13:19:38.473170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:54.085488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"142'424","reported_seq":1788,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.431477+0000","last_change":"2024-09-16T13:19:19.127890+0000","last_active":"2024-09-16T13:19:39.431477+0000","last_peered":"2024-09-16T13:19:39.431477+0000","last_clean":"2024-09-16T13:19:39.431477+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:19:39.431477+0000","last_undegraded":"2024-09-16T13:19:39.431477+0000","last_fullsized":"2024-09-16T13:19:39.431477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020916900000000002,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3126,"num_read_kb":11030,"num_write":475,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773196+0000","last_change":"2024-09-16T13:16:45.128754+0000","last_active":"2024-09-16T13:19:38.773196+0000","last_peered":"2024-09-16T13:19:38.773196+0000","last_clean":"2024-09-16T13:19:38.773196+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:19:38.773196+0000","last_undegraded":"2024-09-16T13:19:38.773196+0000","last_fullsized":"2024-09-16T13:19:38.773196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:13.879629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"139'709","reported_seq":2214,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.431870+0000","last_change":"2024-09-16T13:19:19.164008+0000","last_active":"2024-09-16T13:19:39.431870+0000","last_peered":"2024-09-16T13:19:39.431870+0000","last_clean":"2024-09-16T13:19:39.431870+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:19:39.431870+0000","last_undegraded":"2024-09-16T13:19:39.431870+0000","last_fullsized":"2024-09-16T13:19:39.431870+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017855869999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3881,"num_read_kb":31185,"num_write":963,"num_write_kb":16374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":123,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773578+0000","last_change":"2024-09-16T13:18:44.435672+0000","last_active":"2024-09-16T13:19:38.773578+0000","last_peered":"2024-09-16T13:19:38.773578+0000","last_clean":"2024-09-16T13:19:38.773578+0000","last_became_active":"2024-09-16T13:18:44.435188+0000","last_became_peered":"2024-09-16T13:18:44.435188+0000","last_unstale":"2024-09-16T13:19:38.773578+0000","last_undegraded":"2024-09-16T13:19:38.773578+0000","last_fullsized":"2024-09-16T13:19:38.773578+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:44:35.400164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"143'487","reported_seq":2229,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.339257+0000","last_change":"2024-09-16T13:19:19.131189+0000","last_active":"2024-09-16T13:19:39.339257+0000","last_peered":"2024-09-16T13:19:39.339257+0000","last_clean":"2024-09-16T13:19:39.339257+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:19:39.339257+0000","last_undegraded":"2024-09-16T13:19:39.339257+0000","last_fullsized":"2024-09-16T13:19:39.339257+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003557705,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3512,"num_read_kb":15515,"num_write":504,"num_write_kb":6261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"19'1","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473138+0000","last_change":"2024-09-16T13:16:44.841328+0000","last_active":"2024-09-16T13:19:38.473138+0000","last_peered":"2024-09-16T13:19:38.473138+0000","last_clean":"2024-09-16T13:19:38.473138+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:19:38.473138+0000","last_undegraded":"2024-09-16T13:19:38.473138+0000","last_fullsized":"2024-09-16T13:19:38.473138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:41.796027+0000","last_clean_scrub_stamp":"2024-09-16T13:16:41.796027+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:17.734601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":236,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773593+0000","last_change":"2024-09-16T13:18:44.437802+0000","last_active":"2024-09-16T13:19:38.773593+0000","last_peered":"2024-09-16T13:19:38.773593+0000","last_clean":"2024-09-16T13:19:38.773593+0000","last_became_active":"2024-09-16T13:18:44.437418+0000","last_became_peered":"2024-09-16T13:18:44.437418+0000","last_unstale":"2024-09-16T13:19:38.773593+0000","last_undegraded":"2024-09-16T13:19:38.773593+0000","last_fullsized":"2024-09-16T13:19:38.773593+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:33.765413+0000","last_clean_scrub_stamp":"2024-09-16T13:16:33.765413+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:13:29.273034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"143'606","reported_seq":1310,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.338962+0000","last_change":"2024-09-16T13:19:19.131515+0000","last_active":"2024-09-16T13:19:39.338962+0000","last_peered":"2024-09-16T13:19:39.338962+0000","last_clean":"2024-09-16T13:19:39.338962+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:19:39.338962+0000","last_undegraded":"2024-09-16T13:19:39.338962+0000","last_fullsized":"2024-09-16T13:19:39.338962+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033814320000000002,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2017,"num_read_kb":19372,"num_write":789,"num_write_kb":12171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"142'523","reported_seq":3010,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.432272+0000","last_change":"2024-09-16T13:19:19.128362+0000","last_active":"2024-09-16T13:19:39.432272+0000","last_peered":"2024-09-16T13:19:39.432272+0000","last_clean":"2024-09-16T13:19:39.432272+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:19:39.432272+0000","last_undegraded":"2024-09-16T13:19:39.432272+0000","last_fullsized":"2024-09-16T13:19:39.432272+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001533006,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4183,"num_read_kb":31351,"num_write":445,"num_write_kb":12351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"142'942","reported_seq":3022,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.347467+0000","last_change":"2024-09-16T13:19:38.346887+0000","last_active":"2024-09-16T13:19:38.347467+0000","last_peered":"2024-09-16T13:19:38.347467+0000","last_clean":"2024-09-16T13:19:38.347467+0000","last_became_active":"2024-09-16T13:19:38.343200+0000","last_became_peered":"2024-09-16T13:19:38.343200+0000","last_unstale":"2024-09-16T13:19:38.347467+0000","last_undegraded":"2024-09-16T13:19:38.347467+0000","last_fullsized":"2024-09-16T13:19:38.347467+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:37.001790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025018559999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":4754,"num_read_kb":19969,"num_write":1275,"num_write_kb":12323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"142'1140","reported_seq":4460,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.339255+0000","last_change":"2024-09-16T13:19:19.128947+0000","last_active":"2024-09-16T13:19:39.339255+0000","last_peered":"2024-09-16T13:19:39.339255+0000","last_clean":"2024-09-16T13:19:39.339255+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:19:39.339255+0000","last_undegraded":"2024-09-16T13:19:39.339255+0000","last_fullsized":"2024-09-16T13:19:39.339255+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014023239999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":9870,"num_read_kb":27372,"num_write":1699,"num_write_kb":10648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"143'323","reported_seq":1457,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.473160+0000","last_change":"2024-09-16T13:19:19.135617+0000","last_active":"2024-09-16T13:19:38.473160+0000","last_peered":"2024-09-16T13:19:38.473160+0000","last_clean":"2024-09-16T13:19:38.473160+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:19:38.473160+0000","last_undegraded":"2024-09-16T13:19:38.473160+0000","last_fullsized":"2024-09-16T13:19:38.473160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:24:39.937083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010610159999999999,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":1447,"num_read_kb":15039,"num_write":293,"num_write_kb":7034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"143'389","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774008+0000","last_change":"2024-09-16T13:19:19.175390+0000","last_active":"2024-09-16T13:19:38.774008+0000","last_peered":"2024-09-16T13:19:38.774008+0000","last_clean":"2024-09-16T13:19:38.774008+0000","last_became_active":"2024-09-16T13:18:47.468001+0000","last_became_peered":"2024-09-16T13:18:47.468001+0000","last_unstale":"2024-09-16T13:19:38.774008+0000","last_undegraded":"2024-09-16T13:19:38.774008+0000","last_fullsized":"2024-09-16T13:19:38.774008+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:11:32.886561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018957069999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2747,"num_read_kb":18459,"num_write":343,"num_write_kb":14196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"142'510","reported_seq":7096,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773857+0000","last_change":"2024-09-16T13:19:19.173823+0000","last_active":"2024-09-16T13:19:38.773857+0000","last_peered":"2024-09-16T13:19:38.773857+0000","last_clean":"2024-09-16T13:19:38.773857+0000","last_became_active":"2024-09-16T13:18:47.467028+0000","last_became_peered":"2024-09-16T13:18:47.467028+0000","last_unstale":"2024-09-16T13:19:38.773857+0000","last_undegraded":"2024-09-16T13:19:38.773857+0000","last_fullsized":"2024-09-16T13:19:38.773857+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:03:35.852378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092761899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":8104,"num_read_kb":29095,"num_write":570,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1715710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"142'310","reported_seq":1005,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.774295+0000","last_change":"2024-09-16T13:19:19.173006+0000","last_active":"2024-09-16T13:19:38.774295+0000","last_peered":"2024-09-16T13:19:38.774295+0000","last_clean":"2024-09-16T13:19:38.774295+0000","last_became_active":"2024-09-16T13:18:44.444025+0000","last_became_peered":"2024-09-16T13:18:44.444025+0000","last_unstale":"2024-09-16T13:19:38.774295+0000","last_undegraded":"2024-09-16T13:19:38.774295+0000","last_fullsized":"2024-09-16T13:19:38.774295+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:25:03.521529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056513400000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":692,"num_read_kb":10388,"num_write":284,"num_write_kb":7715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":63719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"142'265","reported_seq":1394,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.346281+0000","last_change":"2024-09-16T13:19:38.345793+0000","last_active":"2024-09-16T13:19:38.346281+0000","last_peered":"2024-09-16T13:19:38.346281+0000","last_clean":"2024-09-16T13:19:38.346281+0000","last_became_active":"2024-09-16T13:19:38.344229+0000","last_became_peered":"2024-09-16T13:19:38.344229+0000","last_unstale":"2024-09-16T13:19:38.346281+0000","last_undegraded":"2024-09-16T13:19:38.346281+0000","last_fullsized":"2024-09-16T13:19:38.346281+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:30:56.026666+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00077869099999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":784,"num_read_kb":16305,"num_write":257,"num_write_kb":11082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"142'556","reported_seq":2476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773502+0000","last_change":"2024-09-16T13:19:19.175226+0000","last_active":"2024-09-16T13:19:38.773502+0000","last_peered":"2024-09-16T13:19:38.773502+0000","last_clean":"2024-09-16T13:19:38.773502+0000","last_became_active":"2024-09-16T13:18:47.462561+0000","last_became_peered":"2024-09-16T13:18:47.462561+0000","last_unstale":"2024-09-16T13:19:38.773502+0000","last_undegraded":"2024-09-16T13:19:38.773502+0000","last_fullsized":"2024-09-16T13:19:38.773502+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.079652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021368229999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3858,"num_read_kb":23843,"num_write":732,"num_write_kb":11756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"141'398","reported_seq":7510,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.431821+0000","last_change":"2024-09-16T13:19:19.164016+0000","last_active":"2024-09-16T13:19:39.431821+0000","last_peered":"2024-09-16T13:19:39.431821+0000","last_clean":"2024-09-16T13:19:39.431821+0000","last_became_active":"2024-09-16T13:18:47.462412+0000","last_became_peered":"2024-09-16T13:18:47.462412+0000","last_unstale":"2024-09-16T13:19:39.431821+0000","last_undegraded":"2024-09-16T13:19:39.431821+0000","last_fullsized":"2024-09-16T13:19:39.431821+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001704484,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":8501,"num_read_kb":35036,"num_write":362,"num_write_kb":9220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2494648,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"141'506","reported_seq":2090,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-16T13:19:38.773579+0000","last_change":"2024-09-16T13:19:19.172960+0000","last_active":"2024-09-16T13:19:38.773579+0000","last_peered":"2024-09-16T13:19:38.773579+0000","last_clean":"2024-09-16T13:19:38.773579+0000","last_became_active":"2024-09-16T13:18:44.443832+0000","last_became_peered":"2024-09-16T13:18:44.443832+0000","last_unstale":"2024-09-16T13:19:38.773579+0000","last_undegraded":"2024-09-16T13:19:38.773579+0000","last_fullsized":"2024-09-16T13:19:38.773579+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:29:45.751077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001469344,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":5023,"num_read_kb":18731,"num_write":523,"num_write_kb":7705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1653183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":218,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":214,"num_read":143401,"num_read_kb":680525,"num_write":19167,"num_write_kb":295695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":44660033,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0},"log_size":16477,"ondisk_log_size":16477,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509484,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311732,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060108,"statfs":{"total":96636764160,"available":96317550592,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.84999999999999998},"min":{"1min":0.47299999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2809999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.499,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1910000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.70899999999999996}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.83799999999999997,"15min":0.82799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.3009999999999999,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.98799999999999999,"15min":0.98199999999999998},"min":{"1min":0.64500000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.6789999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":1.127}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.878,"15min":0.89400000000000002},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.268,"5min":1.476,"15min":1.476},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.97699999999999998,"15min":1.0389999999999999},"min":{"1min":0.63300000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.278,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":1.155}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.0409999999999999,"15min":1.081},"min":{"1min":0.73099999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.5350000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.98599999999999999,"15min":0.999},"min":{"1min":0.65200000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.256,"5min":3.0209999999999999,"15min":3.0209999999999999},"last":1.034}]},{"osd":4,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.96599999999999997,"15min":0.98799999999999999},"min":{"1min":0.53800000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.335,"5min":1.825,"15min":1.825},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0309999999999999,"15min":1.0600000000000001},"min":{"1min":0.56299999999999994,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.347,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.2350000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88100000000000001,"15min":0.90300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.409,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.115,"5min":1.1100000000000001,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":1.006}]},{"osd":7,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":3.286,"5min":3.286,"15min":3.286},"last":0.88200000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509483,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311740,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060100,"statfs":{"total":96636764160,"available":96317542400,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.78300000000000003,"15min":0.80100000000000005},"min":{"1min":0.51600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.927,"15min":1.927},"last":1.0449999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78700000000000003,"15min":0.78300000000000003},"min":{"1min":0.52300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.90000000000000002,"15min":0.91600000000000004},"min":{"1min":0.55000000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.0700000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.82499999999999996,"15min":0.82199999999999995},"min":{"1min":0.59699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.93500000000000005},"min":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0029999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.83599999999999997,"15min":0.84399999999999997},"min":{"1min":0.69699999999999995,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.94699999999999995,"15min":0.95399999999999996},"min":{"1min":0.85999999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.81499999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.96199999999999997},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.4239999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.341,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.1000000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94599999999999995,"15min":0.97999999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.115},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.0609999999999999,"15min":1.0920000000000001},"min":{"1min":0.59899999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.1399999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:19:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.373},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":1.1850000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509484,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311736,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060104,"statfs":{"total":96636764160,"available":96317546496,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.45400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65000000000000002,"15min":0.66300000000000003},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92600000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.76600000000000001},"min":{"1min":0.48399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0660000000000001,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.96999999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.76300000000000001,"15min":0.77100000000000002},"min":{"1min":0.57399999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.88500000000000001},"min":{"1min":0.56399999999999995,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1879999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":1.254}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0209999999999999,"15min":0.98899999999999999},"min":{"1min":0.747,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.7989999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":0.754},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.98999999999999999,"15min":0.96499999999999997},"min":{"1min":0.77500000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3100000000000001,"5min":4.9489999999999998,"15min":4.9489999999999998},"last":1.083}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.99299999999999999,"15min":0.95599999999999996},"min":{"1min":0.71499999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3380000000000001,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.087,"15min":1.091},"min":{"1min":0.86499999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.45,"15min":1.45},"last":1.1850000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0649999999999999,"15min":1.0629999999999999},"min":{"1min":0.69999999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.843,"5min":2.843,"15min":2.843},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.01,"15min":1.01},"min":{"1min":0.80300000000000005,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.94099999999999995}]},{"osd":7,"last update":"Mon Sep 16 13:19:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.996,"5min":0.996,"15min":0.996},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.292},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":1.149}]}]},{"osd":1,"up_from":15,"seq":64424509484,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303428,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068412,"statfs":{"total":96636764160,"available":96326053888,"internally_reserved":0,"allocated":1949696,"data_stored":927051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.911,"5min":3.911,"15min":3.911},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.46600000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9729999999999999,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":0.79600000000000004}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9510000000000001,"5min":2.9510000000000001,"15min":2.9510000000000001},"last":0.72799999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.75800000000000001,"15min":0.72799999999999998},"min":{"1min":0.58199999999999996,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.2089999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":0.621},{"interface":"front","average":{"1min":1.1599999999999999,"5min":0.80500000000000005,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.86899999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.86899999999999999,"15min":0.84399999999999997},"min":{"1min":0.56299999999999994,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.96799999999999997,"15min":0.96699999999999997},"min":{"1min":0.51500000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":0.83399999999999996}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":0.97499999999999998,"15min":0.94099999999999995},"min":{"1min":0.86499999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":5.0919999999999996,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":1.022},{"interface":"front","average":{"1min":1.163,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.61899999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":3.8279999999999998},"last":0.90200000000000002}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.0429999999999999,"15min":1.0089999999999999},"min":{"1min":0.75700000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":1.069},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.04,"15min":1.0269999999999999},"min":{"1min":0.58299999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":4.0460000000000003,"5min":4.0460000000000003,"15min":4.0460000000000003},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.8959999999999999,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":0.98999999999999999}]}]},{"osd":7,"up_from":107,"seq":459561500687,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305528,"kb_used_data":1960,"kb_used_omap":50,"kb_used_meta":303501,"kb_avail":94066312,"statfs":{"total":96636764160,"available":96323903488,"internally_reserved":0,"allocated":2007040,"data_stored":1517381,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":51722,"internal_metadata":310785526},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299264,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072576,"statfs":{"total":96636764160,"available":96330317824,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.77800000000000002,"15min":0.76600000000000001},"min":{"1min":0.57399999999999995,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2929999999999999,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96999999999999997,"15min":0.94799999999999995},"min":{"1min":0.52300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.335,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":0.879}]},{"osd":1,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.82299999999999995,"15min":0.78600000000000003},"min":{"1min":0.54900000000000004,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.331,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.876,"15min":0.86699999999999999},"min":{"1min":0.51500000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.212,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":1.1930000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84599999999999997,"15min":0.86599999999999999},"min":{"1min":0.60799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.1579999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.044,"5min":0.95999999999999996,"15min":0.95899999999999996},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3979999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.0469999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.995,"15min":0.96399999999999997},"min":{"1min":0.59399999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.371,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.018,"5min":0.96599999999999997,"15min":0.94999999999999996},"min":{"1min":0.78700000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.335,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":1.077}]},{"osd":4,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.873,"15min":0.88700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.02,"15min":1.0289999999999999},"min":{"1min":0.68899999999999995,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.1619999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.97299999999999998,"15min":0.98099999999999998},"min":{"1min":0.69099999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.321,"5min":1.637,"15min":1.637},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.103,"5min":1.0569999999999999,"15min":1.073},"min":{"1min":0.82499999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.5980000000000001,"5min":1.661,"15min":1.661},"last":1.121}]},{"osd":7,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.2529999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307580,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064260,"statfs":{"total":96636764160,"available":96321802240,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59699999999999998,"15min":0.60299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85599999999999998,"5min":1.159,"15min":1.159},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.38600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.506,"15min":1.506},"last":0.92600000000000005}]},{"osd":1,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.438,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.05,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.218,"5min":3.298,"15min":3.298},"last":0.48199999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79600000000000004,"15min":0.80700000000000005},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0980000000000001,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.91800000000000004},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.266,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.65500000000000003}]},{"osd":4,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.97999999999999998,"15min":1.014},"min":{"1min":0.61499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.5780000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.05,"5min":1.002,"15min":1},"min":{"1min":0.57299999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.6279999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.96899999999999997}]},{"osd":5,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0569999999999999,"15min":1.081},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.726,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":0.878},{"interface":"front","average":{"1min":1.131,"5min":1.0880000000000001,"15min":1.1060000000000001},"min":{"1min":0.69799999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5029999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.70299999999999996}]},{"osd":6,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.054,"15min":1.069},"min":{"1min":0.67600000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.758,"5min":1.857,"15min":1.857},"last":1.008},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.141,"15min":1.123},"min":{"1min":0.64600000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4079999999999999,"5min":7.0090000000000003,"15min":7.0090000000000003},"last":0.81599999999999995}]},{"osd":7,"last update":"Mon Sep 16 13:19:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":1.048},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.83199999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542187,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303464,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068376,"statfs":{"total":96636764160,"available":96326017024,"internally_reserved":0,"allocated":1990656,"data_stored":1517381,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.60699999999999998,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.501,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.01,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.50900000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.224,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.53400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.109,"5min":1.74,"15min":1.74},"last":0.60499999999999998}]},{"osd":3,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.76200000000000001,"15min":0.77600000000000002},"min":{"1min":0.50900000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.48099999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.276,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.76100000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.92700000000000005,"15min":0.92500000000000004},"min":{"1min":0.73399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.615,"5min":1.871,"15min":1.871},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.71499999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.2809999999999999,"5min":1.903,"15min":1.903},"last":1.1890000000000001}]},{"osd":5,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0269999999999999,"15min":1.0069999999999999},"min":{"1min":0.80100000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.357,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.073,"5min":0.97999999999999998,"15min":0.96299999999999997},"min":{"1min":0.70099999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.93200000000000005}]},{"osd":6,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.044,"15min":1.0409999999999999},"min":{"1min":0.71299999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3879999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.0369999999999999,"15min":1.0269999999999999},"min":{"1min":0.75600000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5149999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.258}]},{"osd":7,"last update":"Mon Sep 16 13:19:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.97399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0}]}} 2024-09-16T13:19:51.165 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.165 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.165 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.166 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.166 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.166 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.166 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.166 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.166 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.166 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.166 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.166 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T13:16:41.796027+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.166 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.167 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T13:16:41.796027+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.167 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.167 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T13:16:41.796027+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.167 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.167 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T13:16:41.796027+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.167 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.167 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.167 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T13:16:41.796027+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.167 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.168 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T13:16:41.796027+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.168 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.168 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T13:16:41.796027+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.168 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.168 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T13:16:41.796027+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.168 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T13:16:33.765413+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.168 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.168 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.168 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.169 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.169 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.169 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.169 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.169 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.169 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.169 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.169 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.169 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:19:51.169 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T13:20:11.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:20:11.475 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:20:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:20:11.490 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-16T13:20:09.500354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":214,"num_read":143467,"num_read_kb":680580,"num_write":19270,"num_write_kb":297325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":45250401,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16733,"ondisk_log_size":16733,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2458560,"kb_used_data":15296,"kb_used_omap":62,"kb_used_meta":2443137,"kb_avail":752516160,"statfs":{"total":773094113280,"available":770576547840,"internally_reserved":0,"allocated":15663104,"data_stored":8596916,"data_compressed":67468,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":63650,"internal_metadata":2501773150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003717"},"pg_stats":[{"pgid":"3.1f","version":"143'370","reported_seq":1981,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.432009+0000","last_change":"2024-09-16T13:19:19.135941+0000","last_active":"2024-09-16T13:19:39.432009+0000","last_peered":"2024-09-16T13:19:39.432009+0000","last_clean":"2024-09-16T13:19:39.432009+0000","last_became_active":"2024-09-16T13:18:03.119735+0000","last_became_peered":"2024-09-16T13:18:03.119735+0000","last_unstale":"2024-09-16T13:19:39.432009+0000","last_undegraded":"2024-09-16T13:19:39.432009+0000","last_fullsized":"2024-09-16T13:19:39.432009+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012744049999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2248,"num_read_kb":16063,"num_write":361,"num_write_kb":5586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":922127,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"162'830","reported_seq":2717,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:49.351578+0000","last_change":"2024-09-16T13:19:49.351553+0000","last_active":"2024-09-16T13:19:49.351578+0000","last_peered":"2024-09-16T13:19:49.351578+0000","last_clean":"2024-09-16T13:19:49.351578+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:19:49.351578+0000","last_undegraded":"2024-09-16T13:19:49.351578+0000","last_fullsized":"2024-09-16T13:19:49.351578+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'830","last_scrub_stamp":"2024-09-16T13:19:49.351459+0000","last_deep_scrub":"162'830","last_deep_scrub_stamp":"2024-09-16T13:19:49.351459+0000","last_clean_scrub_stamp":"2024-09-16T13:19:49.351459+0000","objects_scrubbed":6,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:35.542928+0000","scrub_duration":0.013408455,"objects_trimmed":0,"snaptrim_duration":0.003379118,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":5839,"num_read_kb":7327,"num_write":1002,"num_write_kb":6247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"162'453","reported_seq":2688,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:05.528156+0000","last_change":"2024-09-16T13:20:05.528075+0000","last_active":"2024-09-16T13:20:05.528156+0000","last_peered":"2024-09-16T13:20:05.528156+0000","last_clean":"2024-09-16T13:20:05.528156+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:20:05.528156+0000","last_undegraded":"2024-09-16T13:20:05.528156+0000","last_fullsized":"2024-09-16T13:20:05.528156+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'453","last_scrub_stamp":"2024-09-16T13:20:05.527902+0000","last_deep_scrub":"162'453","last_deep_scrub_stamp":"2024-09-16T13:20:05.527902+0000","last_clean_scrub_stamp":"2024-09-16T13:20:05.527902+0000","objects_scrubbed":7,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:43:53.832692+0000","scrub_duration":0.017214238,"objects_trimmed":0,"snaptrim_duration":0.0014771560000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4778,"num_read_kb":13874,"num_write":452,"num_write_kb":6042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"162'454","reported_seq":2743,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:53.191454+0000","last_change":"2024-09-16T13:19:53.191381+0000","last_active":"2024-09-16T13:19:53.191454+0000","last_peered":"2024-09-16T13:19:53.191454+0000","last_clean":"2024-09-16T13:19:53.191454+0000","last_became_active":"2024-09-16T13:19:38.342560+0000","last_became_peered":"2024-09-16T13:19:38.342560+0000","last_unstale":"2024-09-16T13:19:53.191454+0000","last_undegraded":"2024-09-16T13:19:53.191454+0000","last_fullsized":"2024-09-16T13:19:53.191454+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'454","last_scrub_stamp":"2024-09-16T13:19:53.191159+0000","last_deep_scrub":"162'454","last_deep_scrub_stamp":"2024-09-16T13:19:53.191159+0000","last_clean_scrub_stamp":"2024-09-16T13:19:53.191159+0000","objects_scrubbed":5,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:29:23.248373+0000","scrub_duration":0.012286516000000001,"objects_trimmed":0,"snaptrim_duration":0.00301937,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":2995,"num_read_kb":25210,"num_write":556,"num_write_kb":12813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"141'671","reported_seq":5618,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.431987+0000","last_change":"2024-09-16T13:19:19.164598+0000","last_active":"2024-09-16T13:19:39.431987+0000","last_peered":"2024-09-16T13:19:39.431987+0000","last_clean":"2024-09-16T13:19:39.431987+0000","last_became_active":"2024-09-16T13:19:10.071308+0000","last_became_peered":"2024-09-16T13:19:10.071308+0000","last_unstale":"2024-09-16T13:19:39.431987+0000","last_undegraded":"2024-09-16T13:19:39.431987+0000","last_fullsized":"2024-09-16T13:19:39.431987+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002223587,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6572,"num_read_kb":28113,"num_write":920,"num_write_kb":17343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":3647407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"162'669","reported_seq":7981,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:52.310407+0000","last_change":"2024-09-16T13:19:52.310362+0000","last_active":"2024-09-16T13:19:52.310407+0000","last_peered":"2024-09-16T13:19:52.310407+0000","last_clean":"2024-09-16T13:19:52.310407+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:19:52.310407+0000","last_undegraded":"2024-09-16T13:19:52.310407+0000","last_fullsized":"2024-09-16T13:19:52.310407+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'669","last_scrub_stamp":"2024-09-16T13:19:52.310261+0000","last_deep_scrub":"162'669","last_deep_scrub_stamp":"2024-09-16T13:19:52.310261+0000","last_clean_scrub_stamp":"2024-09-16T13:19:52.310261+0000","objects_scrubbed":8,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:43:51.671205+0000","scrub_duration":0.018943841999999999,"objects_trimmed":0,"snaptrim_duration":0.0028974579999999999,"stat_sum":{"num_bytes":19,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":9256,"num_read_kb":55053,"num_write":857,"num_write_kb":18796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"162'616","reported_seq":5522,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:55.906064+0000","last_change":"2024-09-16T13:19:55.906029+0000","last_active":"2024-09-16T13:19:55.906064+0000","last_peered":"2024-09-16T13:19:55.906064+0000","last_clean":"2024-09-16T13:19:55.906064+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:19:55.906064+0000","last_undegraded":"2024-09-16T13:19:55.906064+0000","last_fullsized":"2024-09-16T13:19:55.906064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'616","last_scrub_stamp":"2024-09-16T13:19:55.905898+0000","last_deep_scrub":"162'616","last_deep_scrub_stamp":"2024-09-16T13:19:55.905898+0000","last_clean_scrub_stamp":"2024-09-16T13:19:55.905898+0000","objects_scrubbed":8,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:52.093712+0000","scrub_duration":0.015599649,"objects_trimmed":0,"snaptrim_duration":0.002453879,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":7637,"num_read_kb":32040,"num_write":812,"num_write_kb":13589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"162'425","reported_seq":2602,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:03.445517+0000","last_change":"2024-09-16T13:20:03.445479+0000","last_active":"2024-09-16T13:20:03.445517+0000","last_peered":"2024-09-16T13:20:03.445517+0000","last_clean":"2024-09-16T13:20:03.445517+0000","last_became_active":"2024-09-16T13:19:10.071112+0000","last_became_peered":"2024-09-16T13:19:10.071112+0000","last_unstale":"2024-09-16T13:20:03.445517+0000","last_undegraded":"2024-09-16T13:20:03.445517+0000","last_fullsized":"2024-09-16T13:20:03.445517+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'425","last_scrub_stamp":"2024-09-16T13:20:03.445335+0000","last_deep_scrub":"162'425","last_deep_scrub_stamp":"2024-09-16T13:20:03.445335+0000","last_clean_scrub_stamp":"2024-09-16T13:20:03.445335+0000","objects_scrubbed":4,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:15.986519+0000","scrub_duration":0.011368593999999999,"objects_trimmed":0,"snaptrim_duration":0.0032828369999999998,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2803,"num_read_kb":8703,"num_write":538,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":228967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"141'573","reported_seq":2388,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.431938+0000","last_change":"2024-09-16T13:19:19.163188+0000","last_active":"2024-09-16T13:19:39.431938+0000","last_peered":"2024-09-16T13:19:39.431938+0000","last_clean":"2024-09-16T13:19:39.431938+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:19:39.431938+0000","last_undegraded":"2024-09-16T13:19:39.431938+0000","last_fullsized":"2024-09-16T13:19:39.431938+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":573,"log_dups_size":0,"ondisk_log_size":573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000869449,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3700,"num_read_kb":13824,"num_write":600,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"162'414","reported_seq":3119,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:01.559923+0000","last_change":"2024-09-16T13:20:01.559890+0000","last_active":"2024-09-16T13:20:01.559923+0000","last_peered":"2024-09-16T13:20:01.559923+0000","last_clean":"2024-09-16T13:20:01.559923+0000","last_became_active":"2024-09-16T13:18:44.441740+0000","last_became_peered":"2024-09-16T13:18:44.441740+0000","last_unstale":"2024-09-16T13:20:01.559923+0000","last_undegraded":"2024-09-16T13:20:01.559923+0000","last_fullsized":"2024-09-16T13:20:01.559923+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'414","last_scrub_stamp":"2024-09-16T13:20:01.559747+0000","last_deep_scrub":"162'414","last_deep_scrub_stamp":"2024-09-16T13:20:01.559747+0000","last_clean_scrub_stamp":"2024-09-16T13:20:01.559747+0000","objects_scrubbed":4,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:50.647933+0000","scrub_duration":0.011032039,"objects_trimmed":0,"snaptrim_duration":0.00082856900000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":5473,"num_read_kb":21309,"num_write":357,"num_write_kb":5655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1185011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"162'235","reported_seq":848,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:56.102862+0000","last_change":"2024-09-16T13:19:56.102759+0000","last_active":"2024-09-16T13:19:56.102862+0000","last_peered":"2024-09-16T13:19:56.102862+0000","last_clean":"2024-09-16T13:19:56.102862+0000","last_became_active":"2024-09-16T13:18:44.435407+0000","last_became_peered":"2024-09-16T13:18:44.435407+0000","last_unstale":"2024-09-16T13:19:56.102862+0000","last_undegraded":"2024-09-16T13:19:56.102862+0000","last_fullsized":"2024-09-16T13:19:56.102862+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'235","last_scrub_stamp":"2024-09-16T13:19:56.102465+0000","last_deep_scrub":"162'235","last_deep_scrub_stamp":"2024-09-16T13:19:56.102465+0000","last_clean_scrub_stamp":"2024-09-16T13:19:56.102465+0000","objects_scrubbed":6,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:54.483268+0000","scrub_duration":0.014244356999999999,"objects_trimmed":0,"snaptrim_duration":0.0016773129999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3249,"num_read_kb":15295,"num_write":190,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1086312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"162'308","reported_seq":5081,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:54.322868+0000","last_change":"2024-09-16T13:19:54.322720+0000","last_active":"2024-09-16T13:19:54.322868+0000","last_peered":"2024-09-16T13:19:54.322868+0000","last_clean":"2024-09-16T13:19:54.322868+0000","last_became_active":"2024-09-16T13:18:47.462532+0000","last_became_peered":"2024-09-16T13:18:47.462532+0000","last_unstale":"2024-09-16T13:19:54.322868+0000","last_undegraded":"2024-09-16T13:19:54.322868+0000","last_fullsized":"2024-09-16T13:19:54.322868+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'308","last_scrub_stamp":"2024-09-16T13:19:54.322479+0000","last_deep_scrub":"162'308","last_deep_scrub_stamp":"2024-09-16T13:19:54.322479+0000","last_clean_scrub_stamp":"2024-09-16T13:19:54.322479+0000","objects_scrubbed":4,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:22:18.166781+0000","scrub_duration":0.010621462999999999,"objects_trimmed":0,"snaptrim_duration":0.002111417,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4736,"num_read_kb":25385,"num_write":347,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1053570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":168,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:55.190618+0000","last_change":"2024-09-16T13:19:55.190524+0000","last_active":"2024-09-16T13:19:55.190618+0000","last_peered":"2024-09-16T13:19:55.190618+0000","last_clean":"2024-09-16T13:19:55.190618+0000","last_became_active":"2024-09-16T13:19:38.775280+0000","last_became_peered":"2024-09-16T13:19:38.775280+0000","last_unstale":"2024-09-16T13:19:55.190618+0000","last_undegraded":"2024-09-16T13:19:55.190618+0000","last_fullsized":"2024-09-16T13:19:55.190618+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:55.190263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:55.190263+0000","last_clean_scrub_stamp":"2024-09-16T13:19:55.190263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:42.014740+0000","scrub_duration":0.0036162809999999998,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"143'692","reported_seq":5655,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.432005+0000","last_change":"2024-09-16T13:19:19.164842+0000","last_active":"2024-09-16T13:19:39.432005+0000","last_peered":"2024-09-16T13:19:39.432005+0000","last_clean":"2024-09-16T13:19:39.432005+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:19:39.432005+0000","last_undegraded":"2024-09-16T13:19:39.432005+0000","last_fullsized":"2024-09-16T13:19:39.432005+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022584039999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6654,"num_read_kb":24625,"num_write":939,"num_write_kb":11031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":165,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:49.844227+0000","last_change":"2024-09-16T13:19:49.844190+0000","last_active":"2024-09-16T13:19:49.844227+0000","last_peered":"2024-09-16T13:19:49.844227+0000","last_clean":"2024-09-16T13:19:49.844227+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:19:49.844227+0000","last_undegraded":"2024-09-16T13:19:49.844227+0000","last_fullsized":"2024-09-16T13:19:49.844227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:49.844048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:49.844048+0000","last_clean_scrub_stamp":"2024-09-16T13:19:49.844048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:20:22.455769+0000","scrub_duration":0.003013497,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"162'312","reported_seq":1492,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:53.451367+0000","last_change":"2024-09-16T13:19:53.451321+0000","last_active":"2024-09-16T13:19:53.451367+0000","last_peered":"2024-09-16T13:19:53.451367+0000","last_clean":"2024-09-16T13:19:53.451367+0000","last_became_active":"2024-09-16T13:18:47.463039+0000","last_became_peered":"2024-09-16T13:18:47.463039+0000","last_unstale":"2024-09-16T13:19:53.451367+0000","last_undegraded":"2024-09-16T13:19:53.451367+0000","last_fullsized":"2024-09-16T13:19:53.451367+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'312","last_scrub_stamp":"2024-09-16T13:19:53.451148+0000","last_deep_scrub":"162'312","last_deep_scrub_stamp":"2024-09-16T13:19:53.451148+0000","last_clean_scrub_stamp":"2024-09-16T13:19:53.451148+0000","objects_scrubbed":9,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:49:47.167241+0000","scrub_duration":0.015981009000000001,"objects_trimmed":0,"snaptrim_duration":0.0026327310000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":1655,"num_read_kb":8730,"num_write":270,"num_write_kb":5747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1061466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:54.090776+0000","last_change":"2024-09-16T13:19:54.090696+0000","last_active":"2024-09-16T13:19:54.090776+0000","last_peered":"2024-09-16T13:19:54.090776+0000","last_clean":"2024-09-16T13:19:54.090776+0000","last_became_active":"2024-09-16T13:18:44.434851+0000","last_became_peered":"2024-09-16T13:18:44.434851+0000","last_unstale":"2024-09-16T13:19:54.090776+0000","last_undegraded":"2024-09-16T13:19:54.090776+0000","last_fullsized":"2024-09-16T13:19:54.090776+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:54.090471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:54.090471+0000","last_clean_scrub_stamp":"2024-09-16T13:19:54.090471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:41:20.265418+0000","scrub_duration":0.0039641490000000001,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"162'377","reported_seq":4703,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:00.011350+0000","last_change":"2024-09-16T13:20:00.011300+0000","last_active":"2024-09-16T13:20:00.011350+0000","last_peered":"2024-09-16T13:20:00.011350+0000","last_clean":"2024-09-16T13:20:00.011350+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:20:00.011350+0000","last_undegraded":"2024-09-16T13:20:00.011350+0000","last_fullsized":"2024-09-16T13:20:00.011350+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'377","last_scrub_stamp":"2024-09-16T13:20:00.011134+0000","last_deep_scrub":"162'377","last_deep_scrub_stamp":"2024-09-16T13:20:00.011134+0000","last_clean_scrub_stamp":"2024-09-16T13:20:00.011134+0000","objects_scrubbed":5,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:07.967425+0000","scrub_duration":0.014457906,"objects_trimmed":0,"snaptrim_duration":0.0032315740000000001,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4693,"num_read_kb":26876,"num_write":422,"num_write_kb":8111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":183,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:52.836055+0000","last_change":"2024-09-16T13:19:52.835993+0000","last_active":"2024-09-16T13:19:52.836055+0000","last_peered":"2024-09-16T13:19:52.836055+0000","last_clean":"2024-09-16T13:19:52.836055+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:19:52.836055+0000","last_undegraded":"2024-09-16T13:19:52.836055+0000","last_fullsized":"2024-09-16T13:19:52.836055+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:52.835742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:52.835742+0000","last_clean_scrub_stamp":"2024-09-16T13:19:52.835742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:48:12.436602+0000","scrub_duration":0.0031974270000000001,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"162'555","reported_seq":3864,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:57.585982+0000","last_change":"2024-09-16T13:19:57.585863+0000","last_active":"2024-09-16T13:19:57.585982+0000","last_peered":"2024-09-16T13:19:57.585982+0000","last_clean":"2024-09-16T13:19:57.585982+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:19:57.585982+0000","last_undegraded":"2024-09-16T13:19:57.585982+0000","last_fullsized":"2024-09-16T13:19:57.585982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'555","last_scrub_stamp":"2024-09-16T13:19:57.585580+0000","last_deep_scrub":"162'555","last_deep_scrub_stamp":"2024-09-16T13:19:57.585580+0000","last_clean_scrub_stamp":"2024-09-16T13:19:57.585580+0000","objects_scrubbed":11,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:15:01.224579+0000","scrub_duration":0.021179129000000001,"objects_trimmed":0,"snaptrim_duration":0.001789768,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5947,"num_read_kb":14133,"num_write":543,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"162'514","reported_seq":2516,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:53.340713+0000","last_change":"2024-09-16T13:19:53.340627+0000","last_active":"2024-09-16T13:19:53.340713+0000","last_peered":"2024-09-16T13:19:53.340713+0000","last_clean":"2024-09-16T13:19:53.340713+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:19:53.340713+0000","last_undegraded":"2024-09-16T13:19:53.340713+0000","last_fullsized":"2024-09-16T13:19:53.340713+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'514","last_scrub_stamp":"2024-09-16T13:19:53.340362+0000","last_deep_scrub":"162'514","last_deep_scrub_stamp":"2024-09-16T13:19:53.340362+0000","last_clean_scrub_stamp":"2024-09-16T13:19:53.340362+0000","objects_scrubbed":8,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:34.829483+0000","scrub_duration":0.021677601000000001,"objects_trimmed":0,"snaptrim_duration":0.003357956,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2667,"num_read_kb":21275,"num_write":487,"num_write_kb":9028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"162'3","reported_seq":168,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:50.609104+0000","last_change":"2024-09-16T13:19:50.609032+0000","last_active":"2024-09-16T13:19:50.609104+0000","last_peered":"2024-09-16T13:19:50.609104+0000","last_clean":"2024-09-16T13:19:50.609104+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:19:50.609104+0000","last_undegraded":"2024-09-16T13:19:50.609104+0000","last_fullsized":"2024-09-16T13:19:50.609104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'3","last_scrub_stamp":"2024-09-16T13:19:50.608788+0000","last_deep_scrub":"162'3","last_deep_scrub_stamp":"2024-09-16T13:19:50.608788+0000","last_clean_scrub_stamp":"2024-09-16T13:19:50.608788+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:01:24.209967+0000","scrub_duration":0.0066277599999999999,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"162'432","reported_seq":1806,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:56.939350+0000","last_change":"2024-09-16T13:19:56.939322+0000","last_active":"2024-09-16T13:19:56.939350+0000","last_peered":"2024-09-16T13:19:56.939350+0000","last_clean":"2024-09-16T13:19:56.939350+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:19:56.939350+0000","last_undegraded":"2024-09-16T13:19:56.939350+0000","last_fullsized":"2024-09-16T13:19:56.939350+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'432","last_scrub_stamp":"2024-09-16T13:19:56.939216+0000","last_deep_scrub":"162'432","last_deep_scrub_stamp":"2024-09-16T13:19:56.939216+0000","last_clean_scrub_stamp":"2024-09-16T13:19:56.939216+0000","objects_scrubbed":8,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:16.773080+0000","scrub_duration":0.014429765000000001,"objects_trimmed":0,"snaptrim_duration":0.0020916900000000002,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3126,"num_read_kb":11030,"num_write":475,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":166,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:49.587922+0000","last_change":"2024-09-16T13:19:49.587888+0000","last_active":"2024-09-16T13:19:49.587922+0000","last_peered":"2024-09-16T13:19:49.587922+0000","last_clean":"2024-09-16T13:19:49.587922+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:19:49.587922+0000","last_undegraded":"2024-09-16T13:19:49.587922+0000","last_fullsized":"2024-09-16T13:19:49.587922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:49.587788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:49.587788+0000","last_clean_scrub_stamp":"2024-09-16T13:19:49.587788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:25:56.343522+0000","scrub_duration":0.0025411930000000002,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"162'717","reported_seq":2232,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:02.767878+0000","last_change":"2024-09-16T13:20:02.767849+0000","last_active":"2024-09-16T13:20:02.767878+0000","last_peered":"2024-09-16T13:20:02.767878+0000","last_clean":"2024-09-16T13:20:02.767878+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:20:02.767878+0000","last_undegraded":"2024-09-16T13:20:02.767878+0000","last_fullsized":"2024-09-16T13:20:02.767878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'717","last_scrub_stamp":"2024-09-16T13:20:02.767731+0000","last_deep_scrub":"162'717","last_deep_scrub_stamp":"2024-09-16T13:20:02.767731+0000","last_clean_scrub_stamp":"2024-09-16T13:20:02.767731+0000","objects_scrubbed":8,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:52.306483+0000","scrub_duration":0.018180992999999999,"objects_trimmed":0,"snaptrim_duration":0.0017855869999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3881,"num_read_kb":31185,"num_write":963,"num_write_kb":16374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:51.073556+0000","last_change":"2024-09-16T13:19:51.073484+0000","last_active":"2024-09-16T13:19:51.073556+0000","last_peered":"2024-09-16T13:19:51.073556+0000","last_clean":"2024-09-16T13:19:51.073556+0000","last_became_active":"2024-09-16T13:18:44.435188+0000","last_became_peered":"2024-09-16T13:18:44.435188+0000","last_unstale":"2024-09-16T13:19:51.073556+0000","last_undegraded":"2024-09-16T13:19:51.073556+0000","last_fullsized":"2024-09-16T13:19:51.073556+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:51.073244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:51.073244+0000","last_clean_scrub_stamp":"2024-09-16T13:19:51.073244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:30:36.470132+0000","scrub_duration":0.0033050430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"162'495","reported_seq":2247,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:55.459769+0000","last_change":"2024-09-16T13:19:55.459739+0000","last_active":"2024-09-16T13:19:55.459769+0000","last_peered":"2024-09-16T13:19:55.459769+0000","last_clean":"2024-09-16T13:19:55.459769+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:19:55.459769+0000","last_undegraded":"2024-09-16T13:19:55.459769+0000","last_fullsized":"2024-09-16T13:19:55.459769+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'495","last_scrub_stamp":"2024-09-16T13:19:55.459651+0000","last_deep_scrub":"162'495","last_deep_scrub_stamp":"2024-09-16T13:19:55.459651+0000","last_clean_scrub_stamp":"2024-09-16T13:19:55.459651+0000","objects_scrubbed":8,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:59:15.365068+0000","scrub_duration":0.014082641999999999,"objects_trimmed":0,"snaptrim_duration":0.003557705,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3512,"num_read_kb":15515,"num_write":504,"num_write_kb":6261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"162'2","reported_seq":167,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:52.654065+0000","last_change":"2024-09-16T13:19:52.653999+0000","last_active":"2024-09-16T13:19:52.654065+0000","last_peered":"2024-09-16T13:19:52.654065+0000","last_clean":"2024-09-16T13:19:52.654065+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:19:52.654065+0000","last_undegraded":"2024-09-16T13:19:52.654065+0000","last_fullsized":"2024-09-16T13:19:52.654065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'2","last_scrub_stamp":"2024-09-16T13:19:52.653770+0000","last_deep_scrub":"162'2","last_deep_scrub_stamp":"2024-09-16T13:19:52.653770+0000","last_clean_scrub_stamp":"2024-09-16T13:19:52.653770+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:39:43.695733+0000","scrub_duration":0.0069047379999999997,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"162'74","reported_seq":249,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:03.126928+0000","last_change":"2024-09-16T13:20:03.126857+0000","last_active":"2024-09-16T13:20:03.126928+0000","last_peered":"2024-09-16T13:20:03.126928+0000","last_clean":"2024-09-16T13:20:03.126928+0000","last_became_active":"2024-09-16T13:18:44.437418+0000","last_became_peered":"2024-09-16T13:18:44.437418+0000","last_unstale":"2024-09-16T13:20:03.126928+0000","last_undegraded":"2024-09-16T13:20:03.126928+0000","last_fullsized":"2024-09-16T13:20:03.126928+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'74","last_scrub_stamp":"2024-09-16T13:20:03.126595+0000","last_deep_scrub":"162'74","last_deep_scrub_stamp":"2024-09-16T13:20:03.126595+0000","last_clean_scrub_stamp":"2024-09-16T13:20:03.126595+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:07:55.651754+0000","scrub_duration":0.013277492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"162'610","reported_seq":1324,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:00.418586+0000","last_change":"2024-09-16T13:20:00.418538+0000","last_active":"2024-09-16T13:20:00.418586+0000","last_peered":"2024-09-16T13:20:00.418586+0000","last_clean":"2024-09-16T13:20:00.418586+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:20:00.418586+0000","last_undegraded":"2024-09-16T13:20:00.418586+0000","last_fullsized":"2024-09-16T13:20:00.418586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'610","last_scrub_stamp":"2024-09-16T13:20:00.418338+0000","last_deep_scrub":"162'610","last_deep_scrub_stamp":"2024-09-16T13:20:00.418338+0000","last_clean_scrub_stamp":"2024-09-16T13:20:00.418338+0000","objects_scrubbed":4,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:49:10.886894+0000","scrub_duration":0.015770830999999999,"objects_trimmed":0,"snaptrim_duration":0.0033814320000000002,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2017,"num_read_kb":19372,"num_write":789,"num_write_kb":12171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"162'528","reported_seq":3025,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:51.323133+0000","last_change":"2024-09-16T13:19:51.323042+0000","last_active":"2024-09-16T13:19:51.323133+0000","last_peered":"2024-09-16T13:19:51.323133+0000","last_clean":"2024-09-16T13:19:51.323133+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:19:51.323133+0000","last_undegraded":"2024-09-16T13:19:51.323133+0000","last_fullsized":"2024-09-16T13:19:51.323133+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'528","last_scrub_stamp":"2024-09-16T13:19:51.322746+0000","last_deep_scrub":"162'528","last_deep_scrub_stamp":"2024-09-16T13:19:51.322746+0000","last_clean_scrub_stamp":"2024-09-16T13:19:51.322746+0000","objects_scrubbed":5,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:24:02.512366+0000","scrub_duration":0.012863693000000001,"objects_trimmed":0,"snaptrim_duration":0.001533006,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4183,"num_read_kb":31351,"num_write":445,"num_write_kb":12351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"162'948","reported_seq":3039,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:51.208878+0000","last_change":"2024-09-16T13:19:51.208850+0000","last_active":"2024-09-16T13:19:51.208878+0000","last_peered":"2024-09-16T13:19:51.208878+0000","last_clean":"2024-09-16T13:19:51.208878+0000","last_became_active":"2024-09-16T13:19:38.343200+0000","last_became_peered":"2024-09-16T13:19:38.343200+0000","last_unstale":"2024-09-16T13:19:51.208878+0000","last_undegraded":"2024-09-16T13:19:51.208878+0000","last_fullsized":"2024-09-16T13:19:51.208878+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'948","last_scrub_stamp":"2024-09-16T13:19:51.208679+0000","last_deep_scrub":"162'948","last_deep_scrub_stamp":"2024-09-16T13:19:51.208679+0000","last_clean_scrub_stamp":"2024-09-16T13:19:51.208679+0000","objects_scrubbed":6,"log_size":948,"log_dups_size":0,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:15:50.936633+0000","scrub_duration":0.015443650999999999,"objects_trimmed":0,"snaptrim_duration":0.0025018559999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":4754,"num_read_kb":19969,"num_write":1275,"num_write_kb":12323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"162'1145","reported_seq":4475,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:51.403223+0000","last_change":"2024-09-16T13:19:51.403192+0000","last_active":"2024-09-16T13:19:51.403223+0000","last_peered":"2024-09-16T13:19:51.403223+0000","last_clean":"2024-09-16T13:19:51.403223+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:19:51.403223+0000","last_undegraded":"2024-09-16T13:19:51.403223+0000","last_fullsized":"2024-09-16T13:19:51.403223+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'1145","last_scrub_stamp":"2024-09-16T13:19:51.403085+0000","last_deep_scrub":"162'1145","last_deep_scrub_stamp":"2024-09-16T13:19:51.403085+0000","last_clean_scrub_stamp":"2024-09-16T13:19:51.403085+0000","objects_scrubbed":5,"log_size":1145,"log_dups_size":0,"ondisk_log_size":1145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:52.199648+0000","scrub_duration":0.014574495999999999,"objects_trimmed":0,"snaptrim_duration":0.0014023239999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":9870,"num_read_kb":27372,"num_write":1699,"num_write_kb":10648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"162'326","reported_seq":1471,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:53.659235+0000","last_change":"2024-09-16T13:19:53.659093+0000","last_active":"2024-09-16T13:19:53.659235+0000","last_peered":"2024-09-16T13:19:53.659235+0000","last_clean":"2024-09-16T13:19:53.659235+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:19:53.659235+0000","last_undegraded":"2024-09-16T13:19:53.659235+0000","last_fullsized":"2024-09-16T13:19:53.659235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'326","last_scrub_stamp":"2024-09-16T13:19:53.658810+0000","last_deep_scrub":"162'326","last_deep_scrub_stamp":"2024-09-16T13:19:53.658810+0000","last_clean_scrub_stamp":"2024-09-16T13:19:53.658810+0000","objects_scrubbed":3,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:56.117307+0000","scrub_duration":0.0098108410000000007,"objects_trimmed":0,"snaptrim_duration":0.0010610159999999999,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":1447,"num_read_kb":15039,"num_write":293,"num_write_kb":7034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"162'402","reported_seq":2500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:06.229630+0000","last_change":"2024-09-16T13:20:06.229578+0000","last_active":"2024-09-16T13:20:06.229630+0000","last_peered":"2024-09-16T13:20:06.229630+0000","last_clean":"2024-09-16T13:20:06.229630+0000","last_became_active":"2024-09-16T13:18:47.468001+0000","last_became_peered":"2024-09-16T13:18:47.468001+0000","last_unstale":"2024-09-16T13:20:06.229630+0000","last_undegraded":"2024-09-16T13:20:06.229630+0000","last_fullsized":"2024-09-16T13:20:06.229630+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'402","last_scrub_stamp":"2024-09-16T13:20:06.229392+0000","last_deep_scrub":"162'402","last_deep_scrub_stamp":"2024-09-16T13:20:06.229392+0000","last_clean_scrub_stamp":"2024-09-16T13:20:06.229392+0000","objects_scrubbed":13,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:20.329610+0000","scrub_duration":0.020004837000000001,"objects_trimmed":0,"snaptrim_duration":0.0018957069999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2747,"num_read_kb":18459,"num_write":343,"num_write_kb":14196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"162'517","reported_seq":7114,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:04.145343+0000","last_change":"2024-09-16T13:20:04.145291+0000","last_active":"2024-09-16T13:20:04.145343+0000","last_peered":"2024-09-16T13:20:04.145343+0000","last_clean":"2024-09-16T13:20:04.145343+0000","last_became_active":"2024-09-16T13:18:47.467028+0000","last_became_peered":"2024-09-16T13:18:47.467028+0000","last_unstale":"2024-09-16T13:20:04.145343+0000","last_undegraded":"2024-09-16T13:20:04.145343+0000","last_fullsized":"2024-09-16T13:20:04.145343+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'517","last_scrub_stamp":"2024-09-16T13:20:04.145120+0000","last_deep_scrub":"162'517","last_deep_scrub_stamp":"2024-09-16T13:20:04.145120+0000","last_clean_scrub_stamp":"2024-09-16T13:20:04.145120+0000","objects_scrubbed":7,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:19:21.749806+0000","scrub_duration":0.018006857000000001,"objects_trimmed":0,"snaptrim_duration":0.00092761899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":8104,"num_read_kb":29095,"num_write":570,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1715710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"162'321","reported_seq":1027,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:00.061951+0000","last_change":"2024-09-16T13:20:00.061911+0000","last_active":"2024-09-16T13:20:00.061951+0000","last_peered":"2024-09-16T13:20:00.061951+0000","last_clean":"2024-09-16T13:20:00.061951+0000","last_became_active":"2024-09-16T13:18:44.444025+0000","last_became_peered":"2024-09-16T13:18:44.444025+0000","last_unstale":"2024-09-16T13:20:00.061951+0000","last_undegraded":"2024-09-16T13:20:00.061951+0000","last_fullsized":"2024-09-16T13:20:00.061951+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'321","last_scrub_stamp":"2024-09-16T13:20:00.061823+0000","last_deep_scrub":"162'321","last_deep_scrub_stamp":"2024-09-16T13:20:00.061823+0000","last_clean_scrub_stamp":"2024-09-16T13:20:00.061823+0000","objects_scrubbed":11,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:05:30.592259+0000","scrub_duration":0.019634558999999999,"objects_trimmed":0,"snaptrim_duration":0.00056513400000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":692,"num_read_kb":10388,"num_write":284,"num_write_kb":7715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":63719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"162'271","reported_seq":1411,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:52.172233+0000","last_change":"2024-09-16T13:19:52.172176+0000","last_active":"2024-09-16T13:19:52.172233+0000","last_peered":"2024-09-16T13:19:52.172233+0000","last_clean":"2024-09-16T13:19:52.172233+0000","last_became_active":"2024-09-16T13:19:38.344229+0000","last_became_peered":"2024-09-16T13:19:38.344229+0000","last_unstale":"2024-09-16T13:19:52.172233+0000","last_undegraded":"2024-09-16T13:19:52.172233+0000","last_fullsized":"2024-09-16T13:19:52.172233+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'271","last_scrub_stamp":"2024-09-16T13:19:52.172048+0000","last_deep_scrub":"162'271","last_deep_scrub_stamp":"2024-09-16T13:19:52.172048+0000","last_clean_scrub_stamp":"2024-09-16T13:19:52.172048+0000","objects_scrubbed":6,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:58.000105+0000","scrub_duration":0.01925456,"objects_trimmed":2,"snaptrim_duration":0.00077869099999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":784,"num_read_kb":16305,"num_write":257,"num_write_kb":11082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"142'556","reported_seq":2477,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:39.433819+0000","last_change":"2024-09-16T13:19:19.175226+0000","last_active":"2024-09-16T13:19:39.433819+0000","last_peered":"2024-09-16T13:19:39.433819+0000","last_clean":"2024-09-16T13:19:39.433819+0000","last_became_active":"2024-09-16T13:18:47.462561+0000","last_became_peered":"2024-09-16T13:18:47.462561+0000","last_unstale":"2024-09-16T13:19:39.433819+0000","last_undegraded":"2024-09-16T13:19:39.433819+0000","last_fullsized":"2024-09-16T13:19:39.433819+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:16:51.271860+0000","last_clean_scrub_stamp":"2024-09-16T13:16:51.271860+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021368229999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3858,"num_read_kb":23843,"num_write":732,"num_write_kb":11756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"162'402","reported_seq":7524,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:00.790358+0000","last_change":"2024-09-16T13:20:00.790285+0000","last_active":"2024-09-16T13:20:00.790358+0000","last_peered":"2024-09-16T13:20:00.790358+0000","last_clean":"2024-09-16T13:20:00.790358+0000","last_became_active":"2024-09-16T13:18:47.462412+0000","last_became_peered":"2024-09-16T13:18:47.462412+0000","last_unstale":"2024-09-16T13:20:00.790358+0000","last_undegraded":"2024-09-16T13:20:00.790358+0000","last_fullsized":"2024-09-16T13:20:00.790358+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'402","last_scrub_stamp":"2024-09-16T13:20:00.790058+0000","last_deep_scrub":"162'402","last_deep_scrub_stamp":"2024-09-16T13:20:00.790058+0000","last_clean_scrub_stamp":"2024-09-16T13:20:00.790058+0000","objects_scrubbed":4,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:49.411889+0000","scrub_duration":0.013079621,"objects_trimmed":0,"snaptrim_duration":0.001704484,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":8501,"num_read_kb":35036,"num_write":362,"num_write_kb":9220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2494648,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"162'516","reported_seq":2111,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:52.093450+0000","last_change":"2024-09-16T13:19:52.093388+0000","last_active":"2024-09-16T13:19:52.093450+0000","last_peered":"2024-09-16T13:19:52.093450+0000","last_clean":"2024-09-16T13:19:52.093450+0000","last_became_active":"2024-09-16T13:18:44.443832+0000","last_became_peered":"2024-09-16T13:18:44.443832+0000","last_unstale":"2024-09-16T13:19:52.093450+0000","last_undegraded":"2024-09-16T13:19:52.093450+0000","last_fullsized":"2024-09-16T13:19:52.093450+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'516","last_scrub_stamp":"2024-09-16T13:19:52.093261+0000","last_deep_scrub":"162'516","last_deep_scrub_stamp":"2024-09-16T13:19:52.093261+0000","last_clean_scrub_stamp":"2024-09-16T13:19:52.093261+0000","objects_scrubbed":10,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:30:10.796502+0000","scrub_duration":0.015993473000000001,"objects_trimmed":0,"snaptrim_duration":0.001469344,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":5023,"num_read_kb":18731,"num_write":523,"num_write_kb":7705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1653183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":218,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":214,"num_read":143401,"num_read_kb":680525,"num_write":19167,"num_write_kb":295695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":44660033,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0},"log_size":16654,"ondisk_log_size":16654,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509488,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311728,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060112,"statfs":{"total":96636764160,"available":96317554688,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.84999999999999998},"min":{"1min":0.47299999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2809999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.499,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1910000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.76100000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.83799999999999997,"15min":0.82799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.3009999999999999,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":1.054},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.98799999999999999,"15min":0.98199999999999998},"min":{"1min":0.64500000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.6789999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":1.081}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.878,"15min":0.89400000000000002},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.268,"5min":1.476,"15min":1.476},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.97699999999999998,"15min":1.0389999999999999},"min":{"1min":0.63300000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.278,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":0.85899999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.0409999999999999,"15min":1.081},"min":{"1min":0.73099999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.5350000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":1.03},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.98599999999999999,"15min":0.999},"min":{"1min":0.65200000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.256,"5min":3.0209999999999999,"15min":3.0209999999999999},"last":1.107}]},{"osd":4,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.96599999999999997,"15min":0.98799999999999999},"min":{"1min":0.53800000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.335,"5min":1.825,"15min":1.825},"last":1.177},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0309999999999999,"15min":1.0600000000000001},"min":{"1min":0.56299999999999994,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.347,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.165}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88100000000000001,"15min":0.90300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.409,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.115,"5min":1.1100000000000001,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":0.95399999999999996}]},{"osd":7,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.141},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":3.286,"5min":3.286,"15min":3.286},"last":0.997}]}]},{"osd":4,"up_from":15,"seq":64424509487,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315892,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94055948,"statfs":{"total":96636764160,"available":96313290752,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.03},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.78300000000000003,"15min":0.80100000000000005},"min":{"1min":0.51600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.927,"15min":1.927},"last":0.67400000000000004}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78700000000000003,"15min":0.78300000000000003},"min":{"1min":0.52300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.90000000000000002,"15min":0.91600000000000004},"min":{"1min":0.55000000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.80200000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.82499999999999996,"15min":0.82199999999999995},"min":{"1min":0.59699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.93500000000000005},"min":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.1439999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.83599999999999997,"15min":0.84399999999999997},"min":{"1min":0.69699999999999995,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.94699999999999995,"15min":0.95399999999999996},"min":{"1min":0.85999999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.98899999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.96199999999999997},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.4239999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.75},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.341,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.0780000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94599999999999995,"15min":0.97999999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.115},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.0609999999999999,"15min":1.0920000000000001},"min":{"1min":0.59899999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.2370000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:19:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":1.2030000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509488,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311728,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060112,"statfs":{"total":96636764160,"available":96317554688,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.45400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.68999999999999995}]},{"osd":1,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65000000000000002,"15min":0.66300000000000003},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92600000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.76600000000000001},"min":{"1min":0.48399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0660000000000001,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.747}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.76300000000000001,"15min":0.77100000000000002},"min":{"1min":0.57399999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.88500000000000001},"min":{"1min":0.56399999999999995,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1879999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.92200000000000004}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0209999999999999,"15min":0.98899999999999999},"min":{"1min":0.747,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.7989999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.98999999999999999,"15min":0.96499999999999997},"min":{"1min":0.77500000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3100000000000001,"5min":4.9489999999999998,"15min":4.9489999999999998},"last":1.0009999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.99299999999999999,"15min":0.95599999999999996},"min":{"1min":0.71499999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3380000000000001,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.087,"15min":1.091},"min":{"1min":0.86499999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.45,"15min":1.45},"last":1.1619999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0649999999999999,"15min":1.0629999999999999},"min":{"1min":0.69999999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.843,"5min":2.843,"15min":2.843},"last":1.087},{"interface":"front","average":{"1min":1.046,"5min":1.01,"15min":1.01},"min":{"1min":0.80300000000000005,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.84599999999999997}]},{"osd":7,"last update":"Mon Sep 16 13:19:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.996,"5min":0.996,"15min":0.996},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":1.147}]}]},{"osd":1,"up_from":15,"seq":64424509488,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303416,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068424,"statfs":{"total":96636764160,"available":96326066176,"internally_reserved":0,"allocated":1949696,"data_stored":927051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.911,"5min":3.911,"15min":3.911},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.46600000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9729999999999999,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":0.90100000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9510000000000001,"5min":2.9510000000000001,"15min":2.9510000000000001},"last":0.74099999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.75800000000000001,"15min":0.72799999999999998},"min":{"1min":0.58199999999999996,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.2089999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.1599999999999999,"5min":0.80500000000000005,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.70199999999999996}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.86899999999999999,"15min":0.84399999999999997},"min":{"1min":0.56299999999999994,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.96799999999999997,"15min":0.96699999999999997},"min":{"1min":0.51500000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":0.93899999999999995}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":0.97499999999999998,"15min":0.94099999999999995},"min":{"1min":0.86499999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":5.0919999999999996,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":1.038},{"interface":"front","average":{"1min":1.163,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.61899999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":3.8279999999999998},"last":1.024}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.0429999999999999,"15min":1.0089999999999999},"min":{"1min":0.75700000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.04,"15min":1.0269999999999999},"min":{"1min":0.58299999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":4.0460000000000003,"5min":4.0460000000000003,"15min":4.0460000000000003},"last":1.0669999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.8959999999999999,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":1.008}]}]},{"osd":7,"up_from":107,"seq":459561500691,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305520,"kb_used_data":1960,"kb_used_omap":50,"kb_used_meta":303501,"kb_avail":94066320,"statfs":{"total":96636764160,"available":96323911680,"internally_reserved":0,"allocated":2007040,"data_stored":1517381,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":51722,"internal_metadata":310785526},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.78800000000000003}]},{"osd":1,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.749},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":1.258}]},{"osd":2,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":1.306}]},{"osd":3,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.95599999999999996}]},{"osd":4,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.84499999999999997}]},{"osd":5,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":1.363},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.226}]},{"osd":6,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.048}]}]},{"osd":5,"up_from":14,"seq":60129542191,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299252,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072588,"statfs":{"total":96636764160,"available":96330330112,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.77800000000000002,"15min":0.76600000000000001},"min":{"1min":0.57399999999999995,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2929999999999999,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96999999999999997,"15min":0.94799999999999995},"min":{"1min":0.52300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.335,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":1.0549999999999999}]},{"osd":1,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.82299999999999995,"15min":0.78600000000000003},"min":{"1min":0.54900000000000004,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.331,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.876,"15min":0.86699999999999999},"min":{"1min":0.51500000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.212,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.86599999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84599999999999997,"15min":0.86599999999999999},"min":{"1min":0.60799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.1579999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.044,"5min":0.95999999999999996,"15min":0.95899999999999996},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3979999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.90900000000000003}]},{"osd":3,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.995,"15min":0.96399999999999997},"min":{"1min":0.59399999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.371,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.018,"5min":0.96599999999999997,"15min":0.94999999999999996},"min":{"1min":0.78700000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.335,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":0.97099999999999997}]},{"osd":4,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.873,"15min":0.88700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":1.204},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.02,"15min":1.0289999999999999},"min":{"1min":0.68899999999999995,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.232}]},{"osd":6,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.97299999999999998,"15min":0.98099999999999998},"min":{"1min":0.69099999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.321,"5min":1.637,"15min":1.637},"last":1.175},{"interface":"front","average":{"1min":1.103,"5min":1.0569999999999999,"15min":1.073},"min":{"1min":0.82499999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.5980000000000001,"5min":1.661,"15min":1.661},"last":1.304}]},{"osd":7,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":1.103},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.2709999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542190,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307568,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064272,"statfs":{"total":96636764160,"available":96321814528,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59699999999999998,"15min":0.60299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85599999999999998,"5min":1.159,"15min":1.159},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.38600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.506,"15min":1.506},"last":0.81499999999999995}]},{"osd":1,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.438,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.05,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.218,"5min":3.298,"15min":3.298},"last":0.57399999999999995}]},{"osd":3,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79600000000000004,"15min":0.80700000000000005},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0980000000000001,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.91800000000000004},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.266,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.89800000000000002}]},{"osd":4,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.97999999999999998,"15min":1.014},"min":{"1min":0.61499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.5780000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.05,"5min":1.002,"15min":1},"min":{"1min":0.57299999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.6279999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.103}]},{"osd":5,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0569999999999999,"15min":1.081},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.726,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.0880000000000001,"15min":1.1060000000000001},"min":{"1min":0.69799999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5029999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.97199999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.054,"15min":1.069},"min":{"1min":0.67600000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.758,"5min":1.857,"15min":1.857},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.141,"15min":1.123},"min":{"1min":0.64600000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4079999999999999,"5min":7.0090000000000003,"15min":7.0090000000000003},"last":1.0509999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:19:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.997}]}]},{"osd":0,"up_from":14,"seq":60129542191,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303456,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068384,"statfs":{"total":96636764160,"available":96326025216,"internally_reserved":0,"allocated":1990656,"data_stored":1517381,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.60699999999999998,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.501,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.01,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996}]},{"osd":2,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.224,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.53400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.109,"5min":1.74,"15min":1.74},"last":0.745}]},{"osd":3,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.76200000000000001,"15min":0.77600000000000002},"min":{"1min":0.50900000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.48099999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.276,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.0820000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.92700000000000005,"15min":0.92500000000000004},"min":{"1min":0.73399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.615,"5min":1.871,"15min":1.871},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.71499999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.2809999999999999,"5min":1.903,"15min":1.903},"last":0.85699999999999998}]},{"osd":5,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0269999999999999,"15min":1.0069999999999999},"min":{"1min":0.80100000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.357,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.073,"5min":0.97999999999999998,"15min":0.96299999999999997},"min":{"1min":0.70099999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.0489999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.044,"15min":1.0409999999999999},"min":{"1min":0.71299999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3879999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.03,"5min":1.0369999999999999,"15min":1.0269999999999999},"min":{"1min":0.75600000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5149999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.022}]},{"osd":7,"last update":"Mon Sep 16 13:19:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0}]}} 2024-09-16T13:20:11.493 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:20:11.493 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:20:11.493 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:20:11.494 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:20:11.494 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-16T13:16:51.271860+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=16, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=19, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:20:11.494 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T13:20:31.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T13:20:31.803 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T13:20:31.806 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T13:20:31.819 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-16T13:20:31.507359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":214,"num_read":143467,"num_read_kb":680580,"num_write":19270,"num_write_kb":297325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":45250401,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16768,"ondisk_log_size":16768,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2458560,"kb_used_data":15296,"kb_used_omap":62,"kb_used_meta":2443137,"kb_avail":752516160,"statfs":{"total":773094113280,"available":770576547840,"internally_reserved":0,"allocated":15663104,"data_stored":8596916,"data_compressed":67468,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":63650,"internal_metadata":2501773150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003960"},"pg_stats":[{"pgid":"3.1f","version":"162'376","reported_seq":1997,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:12.503208+0000","last_change":"2024-09-16T13:20:12.503117+0000","last_active":"2024-09-16T13:20:12.503208+0000","last_peered":"2024-09-16T13:20:12.503208+0000","last_clean":"2024-09-16T13:20:12.503208+0000","last_became_active":"2024-09-16T13:18:03.119735+0000","last_became_peered":"2024-09-16T13:18:03.119735+0000","last_unstale":"2024-09-16T13:20:12.503208+0000","last_undegraded":"2024-09-16T13:20:12.503208+0000","last_fullsized":"2024-09-16T13:20:12.503208+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'376","last_scrub_stamp":"2024-09-16T13:20:12.502798+0000","last_deep_scrub":"162'376","last_deep_scrub_stamp":"2024-09-16T13:20:12.502798+0000","last_clean_scrub_stamp":"2024-09-16T13:20:12.502798+0000","objects_scrubbed":6,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:44.030261+0000","scrub_duration":0.013587079,"objects_trimmed":0,"snaptrim_duration":0.0012744049999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":2248,"num_read_kb":16063,"num_write":361,"num_write_kb":5586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":922127,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"162'830","reported_seq":2717,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:49.351578+0000","last_change":"2024-09-16T13:19:49.351553+0000","last_active":"2024-09-16T13:19:49.351578+0000","last_peered":"2024-09-16T13:19:49.351578+0000","last_clean":"2024-09-16T13:19:49.351578+0000","last_became_active":"2024-09-16T13:16:52.291214+0000","last_became_peered":"2024-09-16T13:16:52.291214+0000","last_unstale":"2024-09-16T13:19:49.351578+0000","last_undegraded":"2024-09-16T13:19:49.351578+0000","last_fullsized":"2024-09-16T13:19:49.351578+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'830","last_scrub_stamp":"2024-09-16T13:19:49.351459+0000","last_deep_scrub":"162'830","last_deep_scrub_stamp":"2024-09-16T13:19:49.351459+0000","last_clean_scrub_stamp":"2024-09-16T13:19:49.351459+0000","objects_scrubbed":6,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:35.542928+0000","scrub_duration":0.013408455,"objects_trimmed":0,"snaptrim_duration":0.003379118,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":5839,"num_read_kb":7327,"num_write":1002,"num_write_kb":6247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"162'453","reported_seq":2688,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:05.528156+0000","last_change":"2024-09-16T13:20:05.528075+0000","last_active":"2024-09-16T13:20:05.528156+0000","last_peered":"2024-09-16T13:20:05.528156+0000","last_clean":"2024-09-16T13:20:05.528156+0000","last_became_active":"2024-09-16T13:16:52.292486+0000","last_became_peered":"2024-09-16T13:16:52.292486+0000","last_unstale":"2024-09-16T13:20:05.528156+0000","last_undegraded":"2024-09-16T13:20:05.528156+0000","last_fullsized":"2024-09-16T13:20:05.528156+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'453","last_scrub_stamp":"2024-09-16T13:20:05.527902+0000","last_deep_scrub":"162'453","last_deep_scrub_stamp":"2024-09-16T13:20:05.527902+0000","last_clean_scrub_stamp":"2024-09-16T13:20:05.527902+0000","objects_scrubbed":7,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:43:53.832692+0000","scrub_duration":0.017214238,"objects_trimmed":0,"snaptrim_duration":0.0014771560000000001,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":4778,"num_read_kb":13874,"num_write":452,"num_write_kb":6042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"162'454","reported_seq":2743,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:53.191454+0000","last_change":"2024-09-16T13:19:53.191381+0000","last_active":"2024-09-16T13:19:53.191454+0000","last_peered":"2024-09-16T13:19:53.191454+0000","last_clean":"2024-09-16T13:19:53.191454+0000","last_became_active":"2024-09-16T13:19:38.342560+0000","last_became_peered":"2024-09-16T13:19:38.342560+0000","last_unstale":"2024-09-16T13:19:53.191454+0000","last_undegraded":"2024-09-16T13:19:53.191454+0000","last_fullsized":"2024-09-16T13:19:53.191454+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'454","last_scrub_stamp":"2024-09-16T13:19:53.191159+0000","last_deep_scrub":"162'454","last_deep_scrub_stamp":"2024-09-16T13:19:53.191159+0000","last_clean_scrub_stamp":"2024-09-16T13:19:53.191159+0000","objects_scrubbed":5,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:29:23.248373+0000","scrub_duration":0.012286516000000001,"objects_trimmed":0,"snaptrim_duration":0.00301937,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":4,"num_read":2995,"num_read_kb":25210,"num_write":556,"num_write_kb":12813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":245046,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"162'681","reported_seq":5638,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:15.006815+0000","last_change":"2024-09-16T13:20:15.006763+0000","last_active":"2024-09-16T13:20:15.006815+0000","last_peered":"2024-09-16T13:20:15.006815+0000","last_clean":"2024-09-16T13:20:15.006815+0000","last_became_active":"2024-09-16T13:19:10.071308+0000","last_became_peered":"2024-09-16T13:19:10.071308+0000","last_unstale":"2024-09-16T13:20:15.006815+0000","last_undegraded":"2024-09-16T13:20:15.006815+0000","last_fullsized":"2024-09-16T13:20:15.006815+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'681","last_scrub_stamp":"2024-09-16T13:20:15.006653+0000","last_deep_scrub":"162'681","last_deep_scrub_stamp":"2024-09-16T13:20:15.006653+0000","last_clean_scrub_stamp":"2024-09-16T13:20:15.006653+0000","objects_scrubbed":10,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:00:44.173089+0000","scrub_duration":0.020207131,"objects_trimmed":0,"snaptrim_duration":0.002223587,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":6572,"num_read_kb":28113,"num_write":920,"num_write_kb":17343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":3647407,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"162'669","reported_seq":7981,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:52.310407+0000","last_change":"2024-09-16T13:19:52.310362+0000","last_active":"2024-09-16T13:19:52.310407+0000","last_peered":"2024-09-16T13:19:52.310407+0000","last_clean":"2024-09-16T13:19:52.310407+0000","last_became_active":"2024-09-16T13:16:52.295894+0000","last_became_peered":"2024-09-16T13:16:52.295894+0000","last_unstale":"2024-09-16T13:19:52.310407+0000","last_undegraded":"2024-09-16T13:19:52.310407+0000","last_fullsized":"2024-09-16T13:19:52.310407+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'669","last_scrub_stamp":"2024-09-16T13:19:52.310261+0000","last_deep_scrub":"162'669","last_deep_scrub_stamp":"2024-09-16T13:19:52.310261+0000","last_clean_scrub_stamp":"2024-09-16T13:19:52.310261+0000","objects_scrubbed":8,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:43:51.671205+0000","scrub_duration":0.018943841999999999,"objects_trimmed":0,"snaptrim_duration":0.0028974579999999999,"stat_sum":{"num_bytes":19,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":9256,"num_read_kb":55053,"num_write":857,"num_write_kb":18796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"162'616","reported_seq":5522,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:55.906064+0000","last_change":"2024-09-16T13:19:55.906029+0000","last_active":"2024-09-16T13:19:55.906064+0000","last_peered":"2024-09-16T13:19:55.906064+0000","last_clean":"2024-09-16T13:19:55.906064+0000","last_became_active":"2024-09-16T13:16:52.294311+0000","last_became_peered":"2024-09-16T13:16:52.294311+0000","last_unstale":"2024-09-16T13:19:55.906064+0000","last_undegraded":"2024-09-16T13:19:55.906064+0000","last_fullsized":"2024-09-16T13:19:55.906064+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'616","last_scrub_stamp":"2024-09-16T13:19:55.905898+0000","last_deep_scrub":"162'616","last_deep_scrub_stamp":"2024-09-16T13:19:55.905898+0000","last_clean_scrub_stamp":"2024-09-16T13:19:55.905898+0000","objects_scrubbed":8,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:52.093712+0000","scrub_duration":0.015599649,"objects_trimmed":0,"snaptrim_duration":0.002453879,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":7637,"num_read_kb":32040,"num_write":812,"num_write_kb":13589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"162'425","reported_seq":2602,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:03.445517+0000","last_change":"2024-09-16T13:20:03.445479+0000","last_active":"2024-09-16T13:20:03.445517+0000","last_peered":"2024-09-16T13:20:03.445517+0000","last_clean":"2024-09-16T13:20:03.445517+0000","last_became_active":"2024-09-16T13:19:10.071112+0000","last_became_peered":"2024-09-16T13:19:10.071112+0000","last_unstale":"2024-09-16T13:20:03.445517+0000","last_undegraded":"2024-09-16T13:20:03.445517+0000","last_fullsized":"2024-09-16T13:20:03.445517+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'425","last_scrub_stamp":"2024-09-16T13:20:03.445335+0000","last_deep_scrub":"162'425","last_deep_scrub_stamp":"2024-09-16T13:20:03.445335+0000","last_clean_scrub_stamp":"2024-09-16T13:20:03.445335+0000","objects_scrubbed":4,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:37:15.986519+0000","scrub_duration":0.011368593999999999,"objects_trimmed":0,"snaptrim_duration":0.0032828369999999998,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2803,"num_read_kb":8703,"num_write":538,"num_write_kb":5489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":228967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"162'579","reported_seq":2404,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:10.977433+0000","last_change":"2024-09-16T13:20:10.977398+0000","last_active":"2024-09-16T13:20:10.977433+0000","last_peered":"2024-09-16T13:20:10.977433+0000","last_clean":"2024-09-16T13:20:10.977433+0000","last_became_active":"2024-09-16T13:17:45.975116+0000","last_became_peered":"2024-09-16T13:17:45.975116+0000","last_unstale":"2024-09-16T13:20:10.977433+0000","last_undegraded":"2024-09-16T13:20:10.977433+0000","last_fullsized":"2024-09-16T13:20:10.977433+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'579","last_scrub_stamp":"2024-09-16T13:20:10.977294+0000","last_deep_scrub":"162'579","last_deep_scrub_stamp":"2024-09-16T13:20:10.977294+0000","last_clean_scrub_stamp":"2024-09-16T13:20:10.977294+0000","objects_scrubbed":6,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:03.039298+0000","scrub_duration":0.014137393,"objects_trimmed":0,"snaptrim_duration":0.000869449,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3700,"num_read_kb":13824,"num_write":600,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":834166,"num_keys_recovered":33,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"162'414","reported_seq":3119,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:01.559923+0000","last_change":"2024-09-16T13:20:01.559890+0000","last_active":"2024-09-16T13:20:01.559923+0000","last_peered":"2024-09-16T13:20:01.559923+0000","last_clean":"2024-09-16T13:20:01.559923+0000","last_became_active":"2024-09-16T13:18:44.441740+0000","last_became_peered":"2024-09-16T13:18:44.441740+0000","last_unstale":"2024-09-16T13:20:01.559923+0000","last_undegraded":"2024-09-16T13:20:01.559923+0000","last_fullsized":"2024-09-16T13:20:01.559923+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'414","last_scrub_stamp":"2024-09-16T13:20:01.559747+0000","last_deep_scrub":"162'414","last_deep_scrub_stamp":"2024-09-16T13:20:01.559747+0000","last_clean_scrub_stamp":"2024-09-16T13:20:01.559747+0000","objects_scrubbed":4,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:50.647933+0000","scrub_duration":0.011032039,"objects_trimmed":0,"snaptrim_duration":0.00082856900000000003,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":5473,"num_read_kb":21309,"num_write":357,"num_write_kb":5655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1185011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"162'235","reported_seq":848,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:56.102862+0000","last_change":"2024-09-16T13:19:56.102759+0000","last_active":"2024-09-16T13:19:56.102862+0000","last_peered":"2024-09-16T13:19:56.102862+0000","last_clean":"2024-09-16T13:19:56.102862+0000","last_became_active":"2024-09-16T13:18:44.435407+0000","last_became_peered":"2024-09-16T13:18:44.435407+0000","last_unstale":"2024-09-16T13:19:56.102862+0000","last_undegraded":"2024-09-16T13:19:56.102862+0000","last_fullsized":"2024-09-16T13:19:56.102862+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'235","last_scrub_stamp":"2024-09-16T13:19:56.102465+0000","last_deep_scrub":"162'235","last_deep_scrub_stamp":"2024-09-16T13:19:56.102465+0000","last_clean_scrub_stamp":"2024-09-16T13:19:56.102465+0000","objects_scrubbed":6,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:54.483268+0000","scrub_duration":0.014244356999999999,"objects_trimmed":0,"snaptrim_duration":0.0016773129999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":3249,"num_read_kb":15295,"num_write":190,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1086312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"162'308","reported_seq":5081,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:54.322868+0000","last_change":"2024-09-16T13:19:54.322720+0000","last_active":"2024-09-16T13:19:54.322868+0000","last_peered":"2024-09-16T13:19:54.322868+0000","last_clean":"2024-09-16T13:19:54.322868+0000","last_became_active":"2024-09-16T13:18:47.462532+0000","last_became_peered":"2024-09-16T13:18:47.462532+0000","last_unstale":"2024-09-16T13:19:54.322868+0000","last_undegraded":"2024-09-16T13:19:54.322868+0000","last_fullsized":"2024-09-16T13:19:54.322868+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'308","last_scrub_stamp":"2024-09-16T13:19:54.322479+0000","last_deep_scrub":"162'308","last_deep_scrub_stamp":"2024-09-16T13:19:54.322479+0000","last_clean_scrub_stamp":"2024-09-16T13:19:54.322479+0000","objects_scrubbed":4,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:22:18.166781+0000","scrub_duration":0.010621462999999999,"objects_trimmed":0,"snaptrim_duration":0.002111417,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":4736,"num_read_kb":25385,"num_write":347,"num_write_kb":7914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1053570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":168,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:55.190618+0000","last_change":"2024-09-16T13:19:55.190524+0000","last_active":"2024-09-16T13:19:55.190618+0000","last_peered":"2024-09-16T13:19:55.190618+0000","last_clean":"2024-09-16T13:19:55.190618+0000","last_became_active":"2024-09-16T13:19:38.775280+0000","last_became_peered":"2024-09-16T13:19:38.775280+0000","last_unstale":"2024-09-16T13:19:55.190618+0000","last_undegraded":"2024-09-16T13:19:55.190618+0000","last_fullsized":"2024-09-16T13:19:55.190618+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:55.190263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:55.190263+0000","last_clean_scrub_stamp":"2024-09-16T13:19:55.190263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:40:42.014740+0000","scrub_duration":0.0036162809999999998,"objects_trimmed":0,"snaptrim_duration":0.00088162699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"162'698","reported_seq":5671,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:09.978241+0000","last_change":"2024-09-16T13:20:09.978211+0000","last_active":"2024-09-16T13:20:09.978241+0000","last_peered":"2024-09-16T13:20:09.978241+0000","last_clean":"2024-09-16T13:20:09.978241+0000","last_became_active":"2024-09-16T13:17:45.977925+0000","last_became_peered":"2024-09-16T13:17:45.977925+0000","last_unstale":"2024-09-16T13:20:09.978241+0000","last_undegraded":"2024-09-16T13:20:09.978241+0000","last_fullsized":"2024-09-16T13:20:09.978241+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'698","last_scrub_stamp":"2024-09-16T13:20:09.978093+0000","last_deep_scrub":"162'698","last_deep_scrub_stamp":"2024-09-16T13:20:09.978093+0000","last_clean_scrub_stamp":"2024-09-16T13:20:09.978093+0000","objects_scrubbed":6,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:11:56.123310+0000","scrub_duration":0.013824461999999999,"objects_trimmed":0,"snaptrim_duration":0.0022584039999999999,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":6654,"num_read_kb":24625,"num_write":939,"num_write_kb":11031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":450908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":165,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:49.844227+0000","last_change":"2024-09-16T13:19:49.844190+0000","last_active":"2024-09-16T13:19:49.844227+0000","last_peered":"2024-09-16T13:19:49.844227+0000","last_clean":"2024-09-16T13:19:49.844227+0000","last_became_active":"2024-09-16T13:16:42.826462+0000","last_became_peered":"2024-09-16T13:16:42.826462+0000","last_unstale":"2024-09-16T13:19:49.844227+0000","last_undegraded":"2024-09-16T13:19:49.844227+0000","last_fullsized":"2024-09-16T13:19:49.844227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:49.844048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:49.844048+0000","last_clean_scrub_stamp":"2024-09-16T13:19:49.844048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:20:22.455769+0000","scrub_duration":0.003013497,"objects_trimmed":0,"snaptrim_duration":0.000739154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"162'312","reported_seq":1492,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:53.451367+0000","last_change":"2024-09-16T13:19:53.451321+0000","last_active":"2024-09-16T13:19:53.451367+0000","last_peered":"2024-09-16T13:19:53.451367+0000","last_clean":"2024-09-16T13:19:53.451367+0000","last_became_active":"2024-09-16T13:18:47.463039+0000","last_became_peered":"2024-09-16T13:18:47.463039+0000","last_unstale":"2024-09-16T13:19:53.451367+0000","last_undegraded":"2024-09-16T13:19:53.451367+0000","last_fullsized":"2024-09-16T13:19:53.451367+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'312","last_scrub_stamp":"2024-09-16T13:19:53.451148+0000","last_deep_scrub":"162'312","last_deep_scrub_stamp":"2024-09-16T13:19:53.451148+0000","last_clean_scrub_stamp":"2024-09-16T13:19:53.451148+0000","objects_scrubbed":9,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:49:47.167241+0000","scrub_duration":0.015981009000000001,"objects_trimmed":0,"snaptrim_duration":0.0026327310000000001,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":9,"num_read":1655,"num_read_kb":8730,"num_write":270,"num_write_kb":5747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1061466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:54.090776+0000","last_change":"2024-09-16T13:19:54.090696+0000","last_active":"2024-09-16T13:19:54.090776+0000","last_peered":"2024-09-16T13:19:54.090776+0000","last_clean":"2024-09-16T13:19:54.090776+0000","last_became_active":"2024-09-16T13:18:44.434851+0000","last_became_peered":"2024-09-16T13:18:44.434851+0000","last_unstale":"2024-09-16T13:19:54.090776+0000","last_undegraded":"2024-09-16T13:19:54.090776+0000","last_fullsized":"2024-09-16T13:19:54.090776+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:54.090471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:54.090471+0000","last_clean_scrub_stamp":"2024-09-16T13:19:54.090471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:41:20.265418+0000","scrub_duration":0.0039641490000000001,"objects_trimmed":0,"snaptrim_duration":0.00080912499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"162'377","reported_seq":4703,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:00.011350+0000","last_change":"2024-09-16T13:20:00.011300+0000","last_active":"2024-09-16T13:20:00.011350+0000","last_peered":"2024-09-16T13:20:00.011350+0000","last_clean":"2024-09-16T13:20:00.011350+0000","last_became_active":"2024-09-16T13:16:52.321755+0000","last_became_peered":"2024-09-16T13:16:52.321755+0000","last_unstale":"2024-09-16T13:20:00.011350+0000","last_undegraded":"2024-09-16T13:20:00.011350+0000","last_fullsized":"2024-09-16T13:20:00.011350+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'377","last_scrub_stamp":"2024-09-16T13:20:00.011134+0000","last_deep_scrub":"162'377","last_deep_scrub_stamp":"2024-09-16T13:20:00.011134+0000","last_clean_scrub_stamp":"2024-09-16T13:20:00.011134+0000","objects_scrubbed":5,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:07.967425+0000","scrub_duration":0.014457906,"objects_trimmed":0,"snaptrim_duration":0.0032315740000000001,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4693,"num_read_kb":26876,"num_write":422,"num_write_kb":8111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":183,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:52.836055+0000","last_change":"2024-09-16T13:19:52.835993+0000","last_active":"2024-09-16T13:19:52.836055+0000","last_peered":"2024-09-16T13:19:52.836055+0000","last_clean":"2024-09-16T13:19:52.836055+0000","last_became_active":"2024-09-16T13:17:42.960033+0000","last_became_peered":"2024-09-16T13:17:42.960033+0000","last_unstale":"2024-09-16T13:19:52.836055+0000","last_undegraded":"2024-09-16T13:19:52.836055+0000","last_fullsized":"2024-09-16T13:19:52.836055+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:52.835742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:52.835742+0000","last_clean_scrub_stamp":"2024-09-16T13:19:52.835742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:48:12.436602+0000","scrub_duration":0.0031974270000000001,"objects_trimmed":0,"snaptrim_duration":0.00080166299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"162'555","reported_seq":3864,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:57.585982+0000","last_change":"2024-09-16T13:19:57.585863+0000","last_active":"2024-09-16T13:19:57.585982+0000","last_peered":"2024-09-16T13:19:57.585982+0000","last_clean":"2024-09-16T13:19:57.585982+0000","last_became_active":"2024-09-16T13:16:52.288599+0000","last_became_peered":"2024-09-16T13:16:52.288599+0000","last_unstale":"2024-09-16T13:19:57.585982+0000","last_undegraded":"2024-09-16T13:19:57.585982+0000","last_fullsized":"2024-09-16T13:19:57.585982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'555","last_scrub_stamp":"2024-09-16T13:19:57.585580+0000","last_deep_scrub":"162'555","last_deep_scrub_stamp":"2024-09-16T13:19:57.585580+0000","last_clean_scrub_stamp":"2024-09-16T13:19:57.585580+0000","objects_scrubbed":11,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:15:01.224579+0000","scrub_duration":0.021179129000000001,"objects_trimmed":0,"snaptrim_duration":0.001789768,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5947,"num_read_kb":14133,"num_write":543,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"162'514","reported_seq":2516,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:53.340713+0000","last_change":"2024-09-16T13:19:53.340627+0000","last_active":"2024-09-16T13:19:53.340713+0000","last_peered":"2024-09-16T13:19:53.340713+0000","last_clean":"2024-09-16T13:19:53.340713+0000","last_became_active":"2024-09-16T13:17:45.974521+0000","last_became_peered":"2024-09-16T13:17:45.974521+0000","last_unstale":"2024-09-16T13:19:53.340713+0000","last_undegraded":"2024-09-16T13:19:53.340713+0000","last_fullsized":"2024-09-16T13:19:53.340713+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'514","last_scrub_stamp":"2024-09-16T13:19:53.340362+0000","last_deep_scrub":"162'514","last_deep_scrub_stamp":"2024-09-16T13:19:53.340362+0000","last_clean_scrub_stamp":"2024-09-16T13:19:53.340362+0000","objects_scrubbed":8,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:34.829483+0000","scrub_duration":0.021677601000000001,"objects_trimmed":0,"snaptrim_duration":0.003357956,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":7,"num_read":2667,"num_read_kb":21275,"num_write":487,"num_write_kb":9028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":674589,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"162'3","reported_seq":168,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:50.609104+0000","last_change":"2024-09-16T13:19:50.609032+0000","last_active":"2024-09-16T13:19:50.609104+0000","last_peered":"2024-09-16T13:19:50.609104+0000","last_clean":"2024-09-16T13:19:50.609104+0000","last_became_active":"2024-09-16T13:16:42.826609+0000","last_became_peered":"2024-09-16T13:16:42.826609+0000","last_unstale":"2024-09-16T13:19:50.609104+0000","last_undegraded":"2024-09-16T13:19:50.609104+0000","last_fullsized":"2024-09-16T13:19:50.609104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'3","last_scrub_stamp":"2024-09-16T13:19:50.608788+0000","last_deep_scrub":"162'3","last_deep_scrub_stamp":"2024-09-16T13:19:50.608788+0000","last_clean_scrub_stamp":"2024-09-16T13:19:50.608788+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:01:24.209967+0000","scrub_duration":0.0066277599999999999,"objects_trimmed":0,"snaptrim_duration":0.0023671209999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"162'432","reported_seq":1806,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:56.939350+0000","last_change":"2024-09-16T13:19:56.939322+0000","last_active":"2024-09-16T13:19:56.939350+0000","last_peered":"2024-09-16T13:19:56.939350+0000","last_clean":"2024-09-16T13:19:56.939350+0000","last_became_active":"2024-09-16T13:16:52.321759+0000","last_became_peered":"2024-09-16T13:16:52.321759+0000","last_unstale":"2024-09-16T13:19:56.939350+0000","last_undegraded":"2024-09-16T13:19:56.939350+0000","last_fullsized":"2024-09-16T13:19:56.939350+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'432","last_scrub_stamp":"2024-09-16T13:19:56.939216+0000","last_deep_scrub":"162'432","last_deep_scrub_stamp":"2024-09-16T13:19:56.939216+0000","last_clean_scrub_stamp":"2024-09-16T13:19:56.939216+0000","objects_scrubbed":8,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:30:16.773080+0000","scrub_duration":0.014429765000000001,"objects_trimmed":0,"snaptrim_duration":0.0020916900000000002,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3126,"num_read_kb":11030,"num_write":475,"num_write_kb":4548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":166,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:49.587922+0000","last_change":"2024-09-16T13:19:49.587888+0000","last_active":"2024-09-16T13:19:49.587922+0000","last_peered":"2024-09-16T13:19:49.587922+0000","last_clean":"2024-09-16T13:19:49.587922+0000","last_became_active":"2024-09-16T13:16:42.824634+0000","last_became_peered":"2024-09-16T13:16:42.824634+0000","last_unstale":"2024-09-16T13:19:49.587922+0000","last_undegraded":"2024-09-16T13:19:49.587922+0000","last_fullsized":"2024-09-16T13:19:49.587922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:49.587788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:49.587788+0000","last_clean_scrub_stamp":"2024-09-16T13:19:49.587788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:25:56.343522+0000","scrub_duration":0.0025411930000000002,"objects_trimmed":0,"snaptrim_duration":0.00048518600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"162'717","reported_seq":2232,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:02.767878+0000","last_change":"2024-09-16T13:20:02.767849+0000","last_active":"2024-09-16T13:20:02.767878+0000","last_peered":"2024-09-16T13:20:02.767878+0000","last_clean":"2024-09-16T13:20:02.767878+0000","last_became_active":"2024-09-16T13:17:45.974235+0000","last_became_peered":"2024-09-16T13:17:45.974235+0000","last_unstale":"2024-09-16T13:20:02.767878+0000","last_undegraded":"2024-09-16T13:20:02.767878+0000","last_fullsized":"2024-09-16T13:20:02.767878+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'717","last_scrub_stamp":"2024-09-16T13:20:02.767731+0000","last_deep_scrub":"162'717","last_deep_scrub_stamp":"2024-09-16T13:20:02.767731+0000","last_clean_scrub_stamp":"2024-09-16T13:20:02.767731+0000","objects_scrubbed":8,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:34:52.306483+0000","scrub_duration":0.018180992999999999,"objects_trimmed":0,"snaptrim_duration":0.0017855869999999999,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3881,"num_read_kb":31185,"num_write":963,"num_write_kb":16374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":13135877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":134,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:51.073556+0000","last_change":"2024-09-16T13:19:51.073484+0000","last_active":"2024-09-16T13:19:51.073556+0000","last_peered":"2024-09-16T13:19:51.073556+0000","last_clean":"2024-09-16T13:19:51.073556+0000","last_became_active":"2024-09-16T13:18:44.435188+0000","last_became_peered":"2024-09-16T13:18:44.435188+0000","last_unstale":"2024-09-16T13:19:51.073556+0000","last_undegraded":"2024-09-16T13:19:51.073556+0000","last_fullsized":"2024-09-16T13:19:51.073556+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:19:51.073244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:19:51.073244+0000","last_clean_scrub_stamp":"2024-09-16T13:19:51.073244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:30:36.470132+0000","scrub_duration":0.0033050430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"162'495","reported_seq":2247,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:55.459769+0000","last_change":"2024-09-16T13:19:55.459739+0000","last_active":"2024-09-16T13:19:55.459769+0000","last_peered":"2024-09-16T13:19:55.459769+0000","last_clean":"2024-09-16T13:19:55.459769+0000","last_became_active":"2024-09-16T13:16:52.294784+0000","last_became_peered":"2024-09-16T13:16:52.294784+0000","last_unstale":"2024-09-16T13:19:55.459769+0000","last_undegraded":"2024-09-16T13:19:55.459769+0000","last_fullsized":"2024-09-16T13:19:55.459769+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'495","last_scrub_stamp":"2024-09-16T13:19:55.459651+0000","last_deep_scrub":"162'495","last_deep_scrub_stamp":"2024-09-16T13:19:55.459651+0000","last_clean_scrub_stamp":"2024-09-16T13:19:55.459651+0000","objects_scrubbed":8,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:59:15.365068+0000","scrub_duration":0.014082641999999999,"objects_trimmed":0,"snaptrim_duration":0.003557705,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":3512,"num_read_kb":15515,"num_write":504,"num_write_kb":6261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"162'2","reported_seq":167,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:52.654065+0000","last_change":"2024-09-16T13:19:52.653999+0000","last_active":"2024-09-16T13:19:52.654065+0000","last_peered":"2024-09-16T13:19:52.654065+0000","last_clean":"2024-09-16T13:19:52.654065+0000","last_became_active":"2024-09-16T13:16:42.827704+0000","last_became_peered":"2024-09-16T13:16:42.827704+0000","last_unstale":"2024-09-16T13:19:52.654065+0000","last_undegraded":"2024-09-16T13:19:52.654065+0000","last_fullsized":"2024-09-16T13:19:52.654065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'2","last_scrub_stamp":"2024-09-16T13:19:52.653770+0000","last_deep_scrub":"162'2","last_deep_scrub_stamp":"2024-09-16T13:19:52.653770+0000","last_clean_scrub_stamp":"2024-09-16T13:19:52.653770+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:39:43.695733+0000","scrub_duration":0.0069047379999999997,"objects_trimmed":0,"snaptrim_duration":0.00065031500000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"162'74","reported_seq":249,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:03.126928+0000","last_change":"2024-09-16T13:20:03.126857+0000","last_active":"2024-09-16T13:20:03.126928+0000","last_peered":"2024-09-16T13:20:03.126928+0000","last_clean":"2024-09-16T13:20:03.126928+0000","last_became_active":"2024-09-16T13:18:44.437418+0000","last_became_peered":"2024-09-16T13:18:44.437418+0000","last_unstale":"2024-09-16T13:20:03.126928+0000","last_undegraded":"2024-09-16T13:20:03.126928+0000","last_fullsized":"2024-09-16T13:20:03.126928+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'74","last_scrub_stamp":"2024-09-16T13:20:03.126595+0000","last_deep_scrub":"162'74","last_deep_scrub_stamp":"2024-09-16T13:20:03.126595+0000","last_clean_scrub_stamp":"2024-09-16T13:20:03.126595+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:07:55.651754+0000","scrub_duration":0.013277492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"162'610","reported_seq":1324,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:00.418586+0000","last_change":"2024-09-16T13:20:00.418538+0000","last_active":"2024-09-16T13:20:00.418586+0000","last_peered":"2024-09-16T13:20:00.418586+0000","last_clean":"2024-09-16T13:20:00.418586+0000","last_became_active":"2024-09-16T13:16:52.292779+0000","last_became_peered":"2024-09-16T13:16:52.292779+0000","last_unstale":"2024-09-16T13:20:00.418586+0000","last_undegraded":"2024-09-16T13:20:00.418586+0000","last_fullsized":"2024-09-16T13:20:00.418586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'610","last_scrub_stamp":"2024-09-16T13:20:00.418338+0000","last_deep_scrub":"162'610","last_deep_scrub_stamp":"2024-09-16T13:20:00.418338+0000","last_clean_scrub_stamp":"2024-09-16T13:20:00.418338+0000","objects_scrubbed":4,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:49:10.886894+0000","scrub_duration":0.015770830999999999,"objects_trimmed":0,"snaptrim_duration":0.0033814320000000002,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":2017,"num_read_kb":19372,"num_write":789,"num_write_kb":12171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"162'528","reported_seq":3025,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:51.323133+0000","last_change":"2024-09-16T13:19:51.323042+0000","last_active":"2024-09-16T13:19:51.323133+0000","last_peered":"2024-09-16T13:19:51.323133+0000","last_clean":"2024-09-16T13:19:51.323133+0000","last_became_active":"2024-09-16T13:16:52.322130+0000","last_became_peered":"2024-09-16T13:16:52.322130+0000","last_unstale":"2024-09-16T13:19:51.323133+0000","last_undegraded":"2024-09-16T13:19:51.323133+0000","last_fullsized":"2024-09-16T13:19:51.323133+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'528","last_scrub_stamp":"2024-09-16T13:19:51.322746+0000","last_deep_scrub":"162'528","last_deep_scrub_stamp":"2024-09-16T13:19:51.322746+0000","last_clean_scrub_stamp":"2024-09-16T13:19:51.322746+0000","objects_scrubbed":5,"log_size":528,"log_dups_size":0,"ondisk_log_size":528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:24:02.512366+0000","scrub_duration":0.012863693000000001,"objects_trimmed":0,"snaptrim_duration":0.001533006,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":4183,"num_read_kb":31351,"num_write":445,"num_write_kb":12351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"162'948","reported_seq":3039,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:51.208878+0000","last_change":"2024-09-16T13:19:51.208850+0000","last_active":"2024-09-16T13:19:51.208878+0000","last_peered":"2024-09-16T13:19:51.208878+0000","last_clean":"2024-09-16T13:19:51.208878+0000","last_became_active":"2024-09-16T13:19:38.343200+0000","last_became_peered":"2024-09-16T13:19:38.343200+0000","last_unstale":"2024-09-16T13:19:51.208878+0000","last_undegraded":"2024-09-16T13:19:51.208878+0000","last_fullsized":"2024-09-16T13:19:51.208878+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'948","last_scrub_stamp":"2024-09-16T13:19:51.208679+0000","last_deep_scrub":"162'948","last_deep_scrub_stamp":"2024-09-16T13:19:51.208679+0000","last_clean_scrub_stamp":"2024-09-16T13:19:51.208679+0000","objects_scrubbed":6,"log_size":948,"log_dups_size":0,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:15:50.936633+0000","scrub_duration":0.015443650999999999,"objects_trimmed":0,"snaptrim_duration":0.0025018559999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":5,"num_read":4754,"num_read_kb":19969,"num_write":1275,"num_write_kb":12323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"162'1145","reported_seq":4475,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:51.403223+0000","last_change":"2024-09-16T13:19:51.403192+0000","last_active":"2024-09-16T13:19:51.403223+0000","last_peered":"2024-09-16T13:19:51.403223+0000","last_clean":"2024-09-16T13:19:51.403223+0000","last_became_active":"2024-09-16T13:17:45.974223+0000","last_became_peered":"2024-09-16T13:17:45.974223+0000","last_unstale":"2024-09-16T13:19:51.403223+0000","last_undegraded":"2024-09-16T13:19:51.403223+0000","last_fullsized":"2024-09-16T13:19:51.403223+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'1145","last_scrub_stamp":"2024-09-16T13:19:51.403085+0000","last_deep_scrub":"162'1145","last_deep_scrub_stamp":"2024-09-16T13:19:51.403085+0000","last_clean_scrub_stamp":"2024-09-16T13:19:51.403085+0000","objects_scrubbed":5,"log_size":1145,"log_dups_size":0,"ondisk_log_size":1145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:40:52.199648+0000","scrub_duration":0.014574495999999999,"objects_trimmed":0,"snaptrim_duration":0.0014023239999999999,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":5,"num_read":9870,"num_read_kb":27372,"num_write":1699,"num_write_kb":10648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":902375,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"162'326","reported_seq":1471,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:53.659235+0000","last_change":"2024-09-16T13:19:53.659093+0000","last_active":"2024-09-16T13:19:53.659235+0000","last_peered":"2024-09-16T13:19:53.659235+0000","last_clean":"2024-09-16T13:19:53.659235+0000","last_became_active":"2024-09-16T13:16:52.290987+0000","last_became_peered":"2024-09-16T13:16:52.290987+0000","last_unstale":"2024-09-16T13:19:53.659235+0000","last_undegraded":"2024-09-16T13:19:53.659235+0000","last_fullsized":"2024-09-16T13:19:53.659235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'326","last_scrub_stamp":"2024-09-16T13:19:53.658810+0000","last_deep_scrub":"162'326","last_deep_scrub_stamp":"2024-09-16T13:19:53.658810+0000","last_clean_scrub_stamp":"2024-09-16T13:19:53.658810+0000","objects_scrubbed":3,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:21:56.117307+0000","scrub_duration":0.0098108410000000007,"objects_trimmed":0,"snaptrim_duration":0.0010610159999999999,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":1447,"num_read_kb":15039,"num_write":293,"num_write_kb":7034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"162'402","reported_seq":2500,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:06.229630+0000","last_change":"2024-09-16T13:20:06.229578+0000","last_active":"2024-09-16T13:20:06.229630+0000","last_peered":"2024-09-16T13:20:06.229630+0000","last_clean":"2024-09-16T13:20:06.229630+0000","last_became_active":"2024-09-16T13:18:47.468001+0000","last_became_peered":"2024-09-16T13:18:47.468001+0000","last_unstale":"2024-09-16T13:20:06.229630+0000","last_undegraded":"2024-09-16T13:20:06.229630+0000","last_fullsized":"2024-09-16T13:20:06.229630+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'402","last_scrub_stamp":"2024-09-16T13:20:06.229392+0000","last_deep_scrub":"162'402","last_deep_scrub_stamp":"2024-09-16T13:20:06.229392+0000","last_clean_scrub_stamp":"2024-09-16T13:20:06.229392+0000","objects_scrubbed":13,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:20.329610+0000","scrub_duration":0.020004837000000001,"objects_trimmed":0,"snaptrim_duration":0.0018957069999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2747,"num_read_kb":18459,"num_write":343,"num_write_kb":14196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":5242880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"162'517","reported_seq":7114,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:04.145343+0000","last_change":"2024-09-16T13:20:04.145291+0000","last_active":"2024-09-16T13:20:04.145343+0000","last_peered":"2024-09-16T13:20:04.145343+0000","last_clean":"2024-09-16T13:20:04.145343+0000","last_became_active":"2024-09-16T13:18:47.467028+0000","last_became_peered":"2024-09-16T13:18:47.467028+0000","last_unstale":"2024-09-16T13:20:04.145343+0000","last_undegraded":"2024-09-16T13:20:04.145343+0000","last_fullsized":"2024-09-16T13:20:04.145343+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'517","last_scrub_stamp":"2024-09-16T13:20:04.145120+0000","last_deep_scrub":"162'517","last_deep_scrub_stamp":"2024-09-16T13:20:04.145120+0000","last_clean_scrub_stamp":"2024-09-16T13:20:04.145120+0000","objects_scrubbed":7,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:19:21.749806+0000","scrub_duration":0.018006857000000001,"objects_trimmed":0,"snaptrim_duration":0.00092761899999999997,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":8104,"num_read_kb":29095,"num_write":570,"num_write_kb":4344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1715710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"162'321","reported_seq":1027,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:00.061951+0000","last_change":"2024-09-16T13:20:00.061911+0000","last_active":"2024-09-16T13:20:00.061951+0000","last_peered":"2024-09-16T13:20:00.061951+0000","last_clean":"2024-09-16T13:20:00.061951+0000","last_became_active":"2024-09-16T13:18:44.444025+0000","last_became_peered":"2024-09-16T13:18:44.444025+0000","last_unstale":"2024-09-16T13:20:00.061951+0000","last_undegraded":"2024-09-16T13:20:00.061951+0000","last_fullsized":"2024-09-16T13:20:00.061951+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'321","last_scrub_stamp":"2024-09-16T13:20:00.061823+0000","last_deep_scrub":"162'321","last_deep_scrub_stamp":"2024-09-16T13:20:00.061823+0000","last_clean_scrub_stamp":"2024-09-16T13:20:00.061823+0000","objects_scrubbed":11,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:05:30.592259+0000","scrub_duration":0.019634558999999999,"objects_trimmed":0,"snaptrim_duration":0.00056513400000000001,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":692,"num_read_kb":10388,"num_write":284,"num_write_kb":7715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":63719,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"162'271","reported_seq":1411,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:52.172233+0000","last_change":"2024-09-16T13:19:52.172176+0000","last_active":"2024-09-16T13:19:52.172233+0000","last_peered":"2024-09-16T13:19:52.172233+0000","last_clean":"2024-09-16T13:19:52.172233+0000","last_became_active":"2024-09-16T13:19:38.344229+0000","last_became_peered":"2024-09-16T13:19:38.344229+0000","last_unstale":"2024-09-16T13:19:52.172233+0000","last_undegraded":"2024-09-16T13:19:52.172233+0000","last_fullsized":"2024-09-16T13:19:52.172233+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'271","last_scrub_stamp":"2024-09-16T13:19:52.172048+0000","last_deep_scrub":"162'271","last_deep_scrub_stamp":"2024-09-16T13:19:52.172048+0000","last_clean_scrub_stamp":"2024-09-16T13:19:52.172048+0000","objects_scrubbed":6,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:48:58.000105+0000","scrub_duration":0.01925456,"objects_trimmed":2,"snaptrim_duration":0.00077869099999999998,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":6,"num_read":784,"num_read_kb":16305,"num_write":257,"num_write_kb":11082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4916344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"162'563","reported_seq":2494,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:13.262080+0000","last_change":"2024-09-16T13:20:13.262047+0000","last_active":"2024-09-16T13:20:13.262080+0000","last_peered":"2024-09-16T13:20:13.262080+0000","last_clean":"2024-09-16T13:20:13.262080+0000","last_became_active":"2024-09-16T13:18:47.462561+0000","last_became_peered":"2024-09-16T13:18:47.462561+0000","last_unstale":"2024-09-16T13:20:13.262080+0000","last_undegraded":"2024-09-16T13:20:13.262080+0000","last_fullsized":"2024-09-16T13:20:13.262080+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'563","last_scrub_stamp":"2024-09-16T13:20:13.261922+0000","last_deep_scrub":"162'563","last_deep_scrub_stamp":"2024-09-16T13:20:13.261922+0000","last_clean_scrub_stamp":"2024-09-16T13:20:13.261922+0000","objects_scrubbed":7,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:47:00.200499+0000","scrub_duration":0.015737164000000001,"objects_trimmed":0,"snaptrim_duration":0.0021368229999999999,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":7,"num_read":3858,"num_read_kb":23843,"num_write":732,"num_write_kb":11756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":3145728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"162'402","reported_seq":7524,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:20:00.790358+0000","last_change":"2024-09-16T13:20:00.790285+0000","last_active":"2024-09-16T13:20:00.790358+0000","last_peered":"2024-09-16T13:20:00.790358+0000","last_clean":"2024-09-16T13:20:00.790358+0000","last_became_active":"2024-09-16T13:18:47.462412+0000","last_became_peered":"2024-09-16T13:18:47.462412+0000","last_unstale":"2024-09-16T13:20:00.790358+0000","last_undegraded":"2024-09-16T13:20:00.790358+0000","last_fullsized":"2024-09-16T13:20:00.790358+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'402","last_scrub_stamp":"2024-09-16T13:20:00.790058+0000","last_deep_scrub":"162'402","last_deep_scrub_stamp":"2024-09-16T13:20:00.790058+0000","last_clean_scrub_stamp":"2024-09-16T13:20:00.790058+0000","objects_scrubbed":4,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:49.411889+0000","scrub_duration":0.013079621,"objects_trimmed":0,"snaptrim_duration":0.001704484,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":4,"num_read":8501,"num_read_kb":35036,"num_write":362,"num_write_kb":9220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":2494648,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"162'516","reported_seq":2111,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-16T13:19:52.093450+0000","last_change":"2024-09-16T13:19:52.093388+0000","last_active":"2024-09-16T13:19:52.093450+0000","last_peered":"2024-09-16T13:19:52.093450+0000","last_clean":"2024-09-16T13:19:52.093450+0000","last_became_active":"2024-09-16T13:18:44.443832+0000","last_became_peered":"2024-09-16T13:18:44.443832+0000","last_unstale":"2024-09-16T13:19:52.093450+0000","last_undegraded":"2024-09-16T13:19:52.093450+0000","last_fullsized":"2024-09-16T13:19:52.093450+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'516","last_scrub_stamp":"2024-09-16T13:19:52.093261+0000","last_deep_scrub":"162'516","last_deep_scrub_stamp":"2024-09-16T13:19:52.093261+0000","last_clean_scrub_stamp":"2024-09-16T13:19:52.093261+0000","objects_scrubbed":10,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:30:10.796502+0000","scrub_duration":0.015993473000000001,"objects_trimmed":0,"snaptrim_duration":0.001469344,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":10,"num_read":5023,"num_read_kb":18731,"num_write":523,"num_write_kb":7705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1653183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":218,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":214,"num_read":143401,"num_read_kb":680525,"num_write":19167,"num_write_kb":295695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":44660033,"num_keys_recovered":148,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0},"log_size":16689,"ondisk_log_size":16689,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":102016,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509492,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311728,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060112,"statfs":{"total":96636764160,"available":96317554688,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82399999999999995,"15min":0.84999999999999998},"min":{"1min":0.47299999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2809999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.3779999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.499,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.1910000000000001,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.0860000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.83799999999999997,"15min":0.82799999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.3009999999999999,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":1.456},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.98799999999999999,"15min":0.98199999999999998},"min":{"1min":0.64500000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.6789999999999998,"5min":2.9169999999999998,"15min":2.9169999999999998},"last":1.6970000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.878,"15min":0.89400000000000002},"min":{"1min":0.68100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.268,"5min":1.476,"15min":1.476},"last":1.1180000000000001},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.97699999999999998,"15min":1.0389999999999999},"min":{"1min":0.63300000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.278,"5min":4.6859999999999999,"15min":4.6859999999999999},"last":0.95599999999999996}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.0409999999999999,"15min":1.081},"min":{"1min":0.73099999999999998,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.5350000000000001,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":1.7929999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.98599999999999999,"15min":0.999},"min":{"1min":0.65200000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.256,"5min":3.0209999999999999,"15min":3.0209999999999999},"last":1.2629999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.96599999999999997,"15min":0.98799999999999999},"min":{"1min":0.53800000000000003,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.335,"5min":1.825,"15min":1.825},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0309999999999999,"15min":1.0600000000000001},"min":{"1min":0.56299999999999994,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.347,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":1.329}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.88100000000000001,"15min":0.90300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.409,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.115,"5min":1.1100000000000001,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.5,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":0.98499999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":3.286,"5min":3.286,"15min":3.286},"last":1.177}]}]},{"osd":4,"up_from":15,"seq":64424509491,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315892,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94055948,"statfs":{"total":96636764160,"available":96313290752,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.80300000000000005,"15min":0.79500000000000004},"min":{"1min":0.46100000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.78300000000000003,"15min":0.80100000000000005},"min":{"1min":0.51600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.282,"5min":1.927,"15min":1.927},"last":1.141}]},{"osd":1,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.78700000000000003,"15min":0.78300000000000003},"min":{"1min":0.52300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.90000000000000002,"15min":0.91600000000000004},"min":{"1min":0.55000000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.96799999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.82499999999999996,"15min":0.82199999999999995},"min":{"1min":0.59699999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89700000000000002,"15min":0.93500000000000005},"min":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1699999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.05}]},{"osd":3,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.83599999999999997,"15min":0.84399999999999997},"min":{"1min":0.69699999999999995,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":1.006},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.94699999999999995,"15min":0.95399999999999996},"min":{"1min":0.85999999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.92400000000000004,"15min":0.96199999999999997},"min":{"1min":0.498,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.4239999999999999,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.202},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.90600000000000003,"15min":0.92000000000000004},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.341,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":0.80600000000000005}]},{"osd":6,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94599999999999995,"15min":0.97999999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.0609999999999999,"15min":1.0920000000000001},"min":{"1min":0.59899999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.85699999999999998}]},{"osd":7,"last update":"Mon Sep 16 13:19:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":1.0820000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509492,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311728,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94060112,"statfs":{"total":96636764160,"available":96317554688,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.45400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.70299999999999996,"15min":0.68999999999999995},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.75}]},{"osd":1,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65000000000000002,"15min":0.66300000000000003},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.92600000000000005,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.76600000000000001},"min":{"1min":0.48399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0660000000000001,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.81699999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.76300000000000001,"15min":0.77100000000000002},"min":{"1min":0.57399999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":1.179},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.86799999999999999,"15min":0.88500000000000001},"min":{"1min":0.56399999999999995,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1879999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.94299999999999995}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0209999999999999,"15min":0.98899999999999999},"min":{"1min":0.747,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.7989999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":1.008},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.98999999999999999,"15min":0.96499999999999997},"min":{"1min":0.77500000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3100000000000001,"5min":4.9489999999999998,"15min":4.9489999999999998},"last":1.0509999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.99299999999999999,"15min":0.95599999999999996},"min":{"1min":0.71499999999999997,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.3380000000000001,"5min":5.3449999999999998,"15min":5.3449999999999998},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.087,"15min":1.091},"min":{"1min":0.86499999999999999,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.444,"5min":1.45,"15min":1.45},"last":1.3999999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.0649999999999999,"15min":1.0629999999999999},"min":{"1min":0.69999999999999996,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.843,"5min":2.843,"15min":2.843},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.046,"5min":1.01,"15min":1.01},"min":{"1min":0.80300000000000005,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.1259999999999999}]},{"osd":7,"last update":"Mon Sep 16 13:19:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.996,"5min":0.996,"15min":0.996},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":1.2130000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509492,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303416,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068424,"statfs":{"total":96636764160,"available":96326066176,"internally_reserved":0,"allocated":1949696,"data_stored":927051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.27800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.911,"5min":3.911,"15min":3.911},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.66000000000000003,"15min":0.61599999999999999},"min":{"1min":0.46600000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.9729999999999999,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.624,"15min":0.59199999999999997},"min":{"1min":0.504,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.117,"5min":4.117,"15min":4.117},"last":0.624},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.34499999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9510000000000001,"5min":2.9510000000000001,"15min":2.9510000000000001},"last":0.752}]},{"osd":3,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.75800000000000001,"15min":0.72799999999999998},"min":{"1min":0.58199999999999996,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":5.2089999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"last":1.075},{"interface":"front","average":{"1min":1.1599999999999999,"5min":0.80500000000000005,"15min":0.74399999999999999},"min":{"1min":0.63800000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":5.0410000000000004},"last":0.81499999999999995}]},{"osd":4,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.86899999999999999,"15min":0.84399999999999997},"min":{"1min":0.56299999999999994,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.7639999999999998,"5min":3.7639999999999998,"15min":3.7639999999999998},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.96799999999999997,"15min":0.96699999999999997},"min":{"1min":0.51500000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":0.90400000000000003}]},{"osd":5,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":0.97499999999999998,"15min":0.94099999999999995},"min":{"1min":0.86499999999999999,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":5.0919999999999996,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.163,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.61899999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":3.8279999999999998},"last":1.0389999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.0429999999999999,"15min":1.0089999999999999},"min":{"1min":0.75700000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.04,"15min":1.0269999999999999},"min":{"1min":0.58299999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":4.0460000000000003,"5min":4.0460000000000003,"15min":4.0460000000000003},"last":1.107}]},{"osd":7,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.8959999999999999,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":0.995}]}]},{"osd":7,"up_from":107,"seq":459561500695,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305520,"kb_used_data":1960,"kb_used_omap":50,"kb_used_meta":303501,"kb_avail":94066320,"statfs":{"total":96636764160,"available":96323911680,"internally_reserved":0,"allocated":2007040,"data_stored":1517381,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":51722,"internal_metadata":310785526},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.68100000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":1.139},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":1.0269999999999999}]},{"osd":3,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":1.073},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.97999999999999998}]},{"osd":4,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":1.1759999999999999},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.0409999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.3080000000000001}]},{"osd":6,"last update":"Mon Sep 16 13:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":1.2310000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542195,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299252,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94072588,"statfs":{"total":96636764160,"available":96330330112,"internally_reserved":0,"allocated":1945600,"data_stored":927032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.77800000000000002,"15min":0.76600000000000001},"min":{"1min":0.57399999999999995,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2929999999999999,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.96999999999999997,"15min":0.94799999999999995},"min":{"1min":0.52300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.335,"5min":9.5120000000000005,"15min":9.5120000000000005},"last":0.83099999999999996}]},{"osd":1,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.82299999999999995,"15min":0.78600000000000003},"min":{"1min":0.54900000000000004,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.331,"5min":2.7370000000000001,"15min":2.7370000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.876,"15min":0.86699999999999999},"min":{"1min":0.51500000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.212,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.78300000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84599999999999997,"15min":0.86599999999999999},"min":{"1min":0.60799999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.1579999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.044,"5min":0.95999999999999996,"15min":0.95899999999999996},"min":{"1min":0.59599999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.3979999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.94799999999999995}]},{"osd":3,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.995,"15min":0.96399999999999997},"min":{"1min":0.59399999999999997,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.371,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.032},{"interface":"front","average":{"1min":1.018,"5min":0.96599999999999997,"15min":0.94999999999999996},"min":{"1min":0.78700000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.335,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":0.86599999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.873,"15min":0.88700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.7370000000000001,"5min":1.7370000000000001,"15min":1.7370000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.02,"15min":1.0289999999999999},"min":{"1min":0.68899999999999995,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.72499999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.97299999999999998,"15min":0.98099999999999998},"min":{"1min":0.69099999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.321,"5min":1.637,"15min":1.637},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.103,"5min":1.0569999999999999,"15min":1.073},"min":{"1min":0.82499999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.5980000000000001,"5min":1.661,"15min":1.661},"last":1.0880000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:19:46 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":1.004}]}]},{"osd":2,"up_from":14,"seq":60129542194,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307568,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94064272,"statfs":{"total":96636764160,"available":96321814528,"internally_reserved":0,"allocated":1941504,"data_stored":927013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59699999999999998,"15min":0.60299999999999998},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.85599999999999998,"5min":1.159,"15min":1.159},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69799999999999995,"15min":0.71799999999999997},"min":{"1min":0.38600000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0649999999999999,"5min":1.506,"15min":1.506},"last":0.93300000000000005}]},{"osd":1,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.76500000000000001,"15min":0.78300000000000003},"min":{"1min":0.438,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.05,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.623},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73099999999999998,"15min":0.72299999999999998},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.218,"5min":3.298,"15min":3.298},"last":0.82099999999999995}]},{"osd":3,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.79600000000000004,"15min":0.80700000000000005},"min":{"1min":0.43099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0980000000000001,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.878,"15min":0.91800000000000004},"min":{"1min":0.502,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.266,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":0.73799999999999999}]},{"osd":4,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.97999999999999998,"15min":1.014},"min":{"1min":0.61499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.5780000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.05,"5min":1.002,"15min":1},"min":{"1min":0.57299999999999995,"5min":0.503,"15min":0.503},"max":{"1min":1.6279999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1}]},{"osd":5,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0569999999999999,"15min":1.081},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.726,"5min":2.7930000000000001,"15min":2.7930000000000001},"last":1.056},{"interface":"front","average":{"1min":1.131,"5min":1.0880000000000001,"15min":1.1060000000000001},"min":{"1min":0.69799999999999995,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.5029999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.97299999999999998}]},{"osd":6,"last update":"Mon Sep 16 13:19:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.054,"15min":1.069},"min":{"1min":0.67600000000000005,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.758,"5min":1.857,"15min":1.857},"last":1.5680000000000001},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.141,"15min":1.123},"min":{"1min":0.64600000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4079999999999999,"5min":7.0090000000000003,"15min":7.0090000000000003},"last":1.081}]},{"osd":7,"last update":"Mon Sep 16 13:19:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.0960000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542195,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303456,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068384,"statfs":{"total":96636764160,"available":96326025216,"internally_reserved":0,"allocated":1990656,"data_stored":1517381,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.60699999999999998,"15min":0.58699999999999997},"min":{"1min":0.433,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.501,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.01,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.76000000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.71899999999999997},"min":{"1min":0.44500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.224,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.53400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.109,"5min":1.74,"15min":1.74},"last":0.879}]},{"osd":3,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.76200000000000001,"15min":0.77600000000000002},"min":{"1min":0.50900000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91900000000000004,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.81799999999999995,"15min":0.82899999999999996},"min":{"1min":0.48099999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.276,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.0740000000000001}]},{"osd":4,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.92700000000000005,"15min":0.92500000000000004},"min":{"1min":0.73399999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.615,"5min":1.871,"15min":1.871},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.71499999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.2809999999999999,"5min":1.903,"15min":1.903},"last":1.0149999999999999}]},{"osd":5,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.0269999999999999,"15min":1.0069999999999999},"min":{"1min":0.80100000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.357,"5min":1.5169999999999999,"15min":1.5169999999999999},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.073,"5min":0.97999999999999998,"15min":0.96299999999999997},"min":{"1min":0.70099999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.2909999999999999}]},{"osd":6,"last update":"Mon Sep 16 13:19:35 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.044,"15min":1.0409999999999999},"min":{"1min":0.71299999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3879999999999999,"5min":1.4490000000000001,"15min":1.4490000000000001},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0369999999999999,"15min":1.0269999999999999},"min":{"1min":0.75600000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.5149999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.1140000000000001}]},{"osd":7,"last update":"Mon Sep 16 13:19:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":1.319},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":1.349}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34548,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32920,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18388,"internal_metadata":0}]}} 2024-09-16T13:20:31.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T13:20:32.140 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T13:20:32.140 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T13:20:32.140 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T13:20:32.140 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.224 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T13:20:32.225 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T13:20:32.225 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.290 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T13:20:32.291 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T13:20:32.291 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.375 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T13:20:32.375 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T13:20:32.375 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.457 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T13:20:32.457 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T13:20:32.457 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.530 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T13:20:32.531 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T13:20:32.531 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.610 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T13:20:32.610 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T13:20:32.611 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.679 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T13:20:32.679 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T13:20:32.679 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.768 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T13:20:32.768 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T13:20:32.768 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T13:20:32.768 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.798 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T13:20:32.798 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T13:20:32.799 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.830 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T13:20:32.830 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T13:20:32.830 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T13:20:32.830 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.847 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T13:20:32.848 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T13:20:32.848 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.907 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T13:20:32.907 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T13:20:32.908 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:20:32.926 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T13:20:32.926 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T13:20:32.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-09-16T13:20:32.946 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:20:32.947 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T13:20:33.175 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:20:33.175 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T13:20:33.379 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:20:33.380 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T13:20:33.554 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:20:33.554 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T13:20:33.715 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:20:33.715 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T13:20:33.964 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:20:33.965 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T13:20:34.213 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:20:34.214 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T13:20:34.418 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:20:34.418 DEBUG:teuthology.orchestra.run.smithi151:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T13:20:34.574 INFO:tasks.ceph:Archiving mon data... 2024-09-16T13:20:34.575 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295/data/mon.a.tgz 2024-09-16T13:20:34.575 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-16T13:20:34.582 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.nzffdsYCge 2024-09-16T13:20:34.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.nzffdsYCge 2024-09-16T13:20:34.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.nzffdsYCge 2024-09-16T13:20:34.706 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.nzffdsYCge is 383KB 2024-09-16T13:20:34.768 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.nzffdsYCge 2024-09-16T13:20:34.774 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295/data/mon.c.tgz 2024-09-16T13:20:34.774 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-16T13:20:34.781 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.ciwtjTm5t7 2024-09-16T13:20:34.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ciwtjTm5t7 2024-09-16T13:20:34.908 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.ciwtjTm5t7 2024-09-16T13:20:34.973 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.ciwtjTm5t7 is 518KB 2024-09-16T13:20:34.996 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.ciwtjTm5t7 2024-09-16T13:20:35.002 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295/data/mon.b.tgz 2024-09-16T13:20:35.002 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-09-16T13:20:35.009 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.EMHwPvz8j6 2024-09-16T13:20:35.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.EMHwPvz8j6 2024-09-16T13:20:35.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.EMHwPvz8j6 2024-09-16T13:20:35.206 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.EMHwPvz8j6 is 518KB 2024-09-16T13:20:35.270 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.EMHwPvz8j6 2024-09-16T13:20:35.277 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T13:20:35.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:20:35.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:20:35.295 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T13:20:35.295 DEBUG:teuthology.orchestra.run.smithi098:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T13:20:35.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T13:20:35.321 INFO:teuthology.orchestra.run.smithi098.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T13:20:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T13:20:35.328 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T13:20:35.330 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295/remote/smithi098/crash 2024-09-16T13:20:35.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T13:20:35.385 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295/remote/smithi151/crash 2024-09-16T13:20:35.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T13:20:35.401 INFO:tasks.ceph:Compressing logs... 2024-09-16T13:20:35.401 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T13:20:35.404 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T13:20:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34334.log 2024-09-16T13:20:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31388.log 2024-09-16T13:20:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32001.log 2024-09-16T13:20:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34334.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.36573.log -- replaced with /var/log/ceph/ceph-client.admin.34334.log.gz 2024-09-16T13:20:35.432 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T13:20:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31388.log: /var/log/ceph/ceph-client.admin.32001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26772.log 2024-09-16T13:20:35.433 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32001.log.gz 2024-09-16T13:20:35.434 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31388.log.gz 2024-09-16T13:20:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36154.log 2024-09-16T13:20:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36573.log.gz 2024-09-16T13:20:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35562.log 2024-09-16T13:20:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26772.log.gz 2024-09-16T13:20:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26070.log 2024-09-16T13:20:35.436 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36154.log.gz 2024-09-16T13:20:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26285.log 2024-09-16T13:20:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35562.log.gz 2024-09-16T13:20:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35756.log 2024-09-16T13:20:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26070.log.gz 2024-09-16T13:20:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28397.log 2024-09-16T13:20:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26285.log: 0.0%/var/log/ceph/ceph-client.admin.35756.log: -- replaced with /var/log/ceph/ceph-client.admin.26285.log.gz 2024-09-16T13:20:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36801.log 2024-09-16T13:20:35.437 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35756.log.gz 2024-09-16T13:20:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31847.log 2024-09-16T13:20:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28397.log.gz 2024-09-16T13:20:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.36801.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36790.log 2024-09-16T13:20:35.439 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36801.log.gz 2024-09-16T13:20:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37430.log 2024-09-16T13:20:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31847.log.gz 2024-09-16T13:20:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26279.log 2024-09-16T13:20:35.439 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36790.log.gz 2024-09-16T13:20:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33250.log 2024-09-16T13:20:35.440 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37430.log.gz 2024-09-16T13:20:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30729.log 2024-09-16T13:20:35.440 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26279.log.gz 2024-09-16T13:20:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32246.log 2024-09-16T13:20:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33250.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.33250.log.gz/var/log/ceph/ceph-client.admin.37052.log 2024-09-16T13:20:35.441 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T13:20:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30729.log.gz 2024-09-16T13:20:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32246.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T13:20:35.442 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32246.log.gz 2024-09-16T13:20:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31165.log 2024-09-16T13:20:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37052.log.gz 2024-09-16T13:20:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.36574.log 2024-09-16T13:20:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29547.log 2024-09-16T13:20:35.443 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31165.log.gz 2024-09-16T13:20:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36026.log 2024-09-16T13:20:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36574.log: /var/log/ceph/ceph-client.admin.29547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36574.log.gz 2024-09-16T13:20:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29547.log.gz/var/log/ceph/ceph-client.admin.27346.log 2024-09-16T13:20:35.443 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T13:20:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37322.log 2024-09-16T13:20:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36026.log: /var/log/ceph/ceph-client.admin.27346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27346.log.gz 2024-09-16T13:20:35.445 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36026.log.gz 2024-09-16T13:20:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37592.log 2024-09-16T13:20:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30309.log 2024-09-16T13:20:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37322.log.gz 2024-09-16T13:20:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26799.log 2024-09-16T13:20:35.445 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37592.log.gz 2024-09-16T13:20:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30309.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30309.log.gz 2024-09-16T13:20:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.37133.log 2024-09-16T13:20:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26799.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.26799.log.gz-- 2024-09-16T13:20:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29522.log 2024-09-16T13:20:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37133.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37133.log.gz 2024-09-16T13:20:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.34625.log 2024-09-16T13:20:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35729.log 2024-09-16T13:20:35.448 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29522.log.gz 2024-09-16T13:20:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34625.log.gz 2024-09-16T13:20:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29298.log 2024-09-16T13:20:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22105.log 2024-09-16T13:20:35.449 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35729.log.gz 2024-09-16T13:20:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36815.log 2024-09-16T13:20:35.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29298.log.gz 2024-09-16T13:20:35.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.22105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35891.log 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22105.log.gz 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35972.log 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36815.log.gz 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37025.log 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35891.log.gz 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35972.log.gz 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35702.log 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28678.log 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37025.log.gz 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34399.log 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35702.log.gz 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28678.log.gz 2024-09-16T13:20:35.456 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26826.log 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36592.log 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34399.log.gz 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37268.log 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26826.log.gz 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28884.log 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36592.log.gz 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27750.log /var/log/ceph/ceph-client.admin.37268.log: 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37268.log.gz 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30970.log 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28884.log.gz 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28106.log 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27750.log.gz 2024-09-16T13:20:35.457 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30970.log: /var/log/ceph/ceph-client.admin.37673.log 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30970.log.gz 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28363.log 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28106.log.gz 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37565.log 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37673.log.gz 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37241.log 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28363.log.gz 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29350.log 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37565.log.gz 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.37241.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36590.log 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37241.log.gz 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26067.log 2024-09-16T13:20:35.458 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29350.log.gz 2024-09-16T13:20:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32565.log 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36590.log.gz 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30363.log 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26067.log.gz 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26926.log 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32565.log.gz 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34461.log 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30363.log.gz 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34041.log 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26926.log.gz 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 /var/log/ceph/ceph-client.admin.34461.log: --verbose -- /var/log/ceph/ceph-client.admin.37214.log 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34461.log.gz 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34041.log.gz 2024-09-16T13:20:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30116.log 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37106.log 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37214.log.gz 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.30116.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32315.log 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30116.log.gz 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36369.log 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37106.log.gz 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26745.log 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32315.log.gz 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27908.log 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36369.log.gz 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26745.log.gz 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27275.log 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29175.log 2024-09-16T13:20:35.467 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27908.log.gz 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36370.log 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27275.log.gz 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29706.log 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29175.log.gz 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34470.log 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36370.log.gz 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29706.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.29706.log.gz--verbose 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:-- /var/log/ceph/ceph-client.admin.29416.log 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28232.log 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34470.log.gz 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29843.log 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29416.log.gz 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30760.log 2024-09-16T13:20:35.468 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28232.log.gz 2024-09-16T13:20:35.469 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29843.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T13:20:35.469 INFO:teuthology.orchestra.run.smithi098.stderr: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.29843.log.gz 2024-09-16T13:20:35.469 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30760.log.gz 2024-09-16T13:20:35.469 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34598.log 2024-09-16T13:20:35.481 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T13:20:35.481 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22138.log 2024-09-16T13:20:35.481 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22112.log 2024-09-16T13:20:35.481 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.28211.log 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.22138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22138.log.gz 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.22112.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool..log 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22112.log.gz 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22248.log 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/objectstore_tool.28211.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/objectstore_tool..log: /var/log/ceph/ceph-client.admin.22248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22192.log 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22248.log.gz 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.27375.log 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.22192.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T13:20:35.482 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-mgr.x.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22192.log.gz 2024-09-16T13:20:35.488 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32529.log 2024-09-16T13:20:35.488 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28483.log 2024-09-16T13:20:35.488 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34598.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34598.log.gz 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29897.log 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32529.log.gz 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35039.log 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28483.log.gz 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36355.log /var/log/ceph/ceph-client.admin.29897.log: 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29897.log.gz 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36162.log 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35039.log: gzip /var/log/ceph/ceph-client.admin.36355.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33303.log 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35039.log.gz 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36355.log.gz 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33178.log 2024-09-16T13:20:35.489 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36162.log.gz 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33303.log.gz 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29772.log 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32125.log 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33178.log.gz 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28512.log 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29772.log.gz 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35783.log 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32125.log: 93.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32125.log.gz 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35616.log 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28512.log.gz 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35783.log: /var/log/ceph/ceph-client.admin.35406.log 2024-09-16T13:20:35.490 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35783.log.gz 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34564.log 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35616.log.gz 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.35406.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26718.log 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35406.log.gz 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36358.log 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34564.log: /var/log/ceph/ceph-client.admin.26718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34564.log.gz 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26718.log.gz 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27972.log 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37295.log 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36358.log.gz 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33676.log 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27972.log: /var/log/ceph/ceph-client.admin.37295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27972.log.gz 2024-09-16T13:20:35.491 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28001.log 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37295.log.gz 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26062.log 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33676.log.gz 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35999.log 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28001.log.gz 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26062.log.gz 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30928.log 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31561.log 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35999.log: /var/log/ceph/ceph-client.admin.30928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35999.log.gz 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30928.log.gz 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25962.log 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31112.log 2024-09-16T13:20:35.492 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31561.log.gz 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35045.log 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.25962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25962.log.gz 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28424.log 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31112.log.gz 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29868.log 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35045.log.gz 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28227.log 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28424.log.gz 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32667.log 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29868.log.gz 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31282.log 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28227.log.gz 2024-09-16T13:20:35.493 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29664.log 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32667.log.gz 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31282.log: /var/log/ceph/ceph-mon.c.log 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31282.log.gz 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35144.log 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29664.log.gz 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37187.log 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35144.log.gz 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37619.log 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.37187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37700.log 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37187.log.gz 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30619.log 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37619.log.gz 2024-09-16T13:20:35.494 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32474.log 2024-09-16T13:20:35.495 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37700.log.gz 2024-09-16T13:20:35.495 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30619.log: /var/log/ceph/ceph-client.admin.35945.log 2024-09-16T13:20:35.495 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30619.log.gz 2024-09-16T13:20:35.495 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27947.log 2024-09-16T13:20:35.495 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32474.log.gz 2024-09-16T13:20:35.495 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31428.log 2024-09-16T13:20:35.495 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35945.log.gz 2024-09-16T13:20:35.495 INFO:teuthology.orchestra.run.smithi098.stderr: 93.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T13:20:35.495 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28137.log 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27947.log.gz 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29021.log 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31428.log.gz 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28137.log: /var/log/ceph/ceph-client.admin.28829.log 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28137.log.gz 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29021.log.gz 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36153.log 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27248.log 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28829.log.gz 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30149.log 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36153.log.gz 2024-09-16T13:20:35.496 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27248.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17295.log 2024-09-16T13:20:35.497 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27248.log.gz 2024-09-16T13:20:35.497 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31887.log 2024-09-16T13:20:35.497 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30149.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30149.log.gz 2024-09-16T13:20:35.497 INFO:teuthology.orchestra.run.smithi098.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.31614.log 2024-09-16T13:20:35.497 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.tmp-client.admin.17295.log: 0.0%/var/log/ceph/ceph-client.admin.31887.log: -- replaced with /var/log/ceph/ceph.tmp-client.admin.17295.log.gz 2024-09-16T13:20:35.497 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35864.log 2024-09-16T13:20:35.497 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31887.log.gz 2024-09-16T13:20:35.504 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/objectstore_tool.27375.log: 95.2% -- replaced with /var/log/ceph/objectstore_tool.28211.log.gz 2024-09-16T13:20:35.504 INFO:teuthology.orchestra.run.smithi151.stderr:gzip 95.4% -- replaced with /var/log/ceph/objectstore_tool.27375.log.gz 2024-09-16T13:20:35.504 INFO:teuthology.orchestra.run.smithi151.stderr:-5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.28976.log 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.29748.log 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/objectstore_tool.28976.log: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22220.log 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/objectstore_tool.29748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22165.log 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.22220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22220.log.gz 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr: 95.3% -- replaced with /var/log/ceph/objectstore_tool..log.gz 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22062.log 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.22165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22165.log.gz 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.22062.log: 95.3% -- replaced with /var/log/ceph/objectstore_tool.28976.log.gz 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22062.log.gz 2024-09-16T13:20:35.505 INFO:teuthology.orchestra.run.smithi151.stderr: 95.3% -- replaced with /var/log/ceph/objectstore_tool.29748.log.gz 2024-09-16T13:20:35.506 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22087.log 2024-09-16T13:20:35.506 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T13:20:35.506 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T13:20:35.506 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-client.admin.22087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22087.log.gz 2024-09-16T13:20:35.506 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-osd.7.log: 83.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T13:20:35.541 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32735.log 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31614.log: 0.0%/var/log/ceph/ceph-client.admin.35864.log: -- replaced with /var/log/ceph/ceph-client.admin.31614.log.gz 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35864.log.gz 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37725.log 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.32735.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33930.log 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32735.log.gz 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34859.log 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37725.log.gz 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.33930.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27571.log 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33930.log.gz 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36998.log 2024-09-16T13:20:35.542 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34859.log.gz 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25916.log 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27571.log.gz 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26488.log 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36998.log.gz 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37457.log 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.25916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25916.log.gz 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28326.log 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26488.log.gz 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31257.log 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37457.log.gz 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36147.log 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28326.log.gz 2024-09-16T13:20:35.543 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27695.log 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31257.log.gz 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36378.log 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36147.log.gz 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29379.log 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27695.log.gz 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26482.log 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36378.log.gz 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26061.log 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29379.log.gz 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26491.log 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26482.log.gz 2024-09-16T13:20:35.544 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36789.log 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26061.log.gz 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33735.log 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26491.log: 0.0%/var/log/ceph/ceph-client.admin.36789.log: -- replaced with /var/log/ceph/ceph-client.admin.26491.log.gz 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32059.log 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36789.log.gz 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33893.log 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33735.log.gz 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26277.log 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32059.log.gz 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28927.log 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33893.log.gz 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27812.log 2024-09-16T13:20:35.545 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26277.log.gz 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34301.log 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28927.log.gz 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26853.log 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27812.log.gz 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35918.log 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34301.log.gz 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26853.log.gz 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35810.log 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35918.log.gz 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26053.log 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.35810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37538.log 2024-09-16T13:20:35.546 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35810.log.gz 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26476.log 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26053.log.gz 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25989.log 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37538.log: /var/log/ceph/ceph-client.admin.26476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35259.log 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26476.log.gz 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37538.log.gz 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36160.log 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.25989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25989.log.gz 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36802.log 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35259.log.gz 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29221.log 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36160.log.gz 2024-09-16T13:20:35.547 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26278.log 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36802.log.gz 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36053.log 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29221.log: /var/log/ceph/ceph-client.admin.26278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30788.log 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29221.log.gz 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26278.log.gz 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33544.log 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36053.log.gz 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34371.log 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30788.log.gz 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30700.log 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33544.log.gz 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36362.log 2024-09-16T13:20:35.548 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34371.log.gz 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.30700.log: -- /var/log/ceph/ceph-client.admin.31895.log 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30700.log.gz 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26499.log 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36362.log.gz 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34703.log 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31895.log.gz 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.26499.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36813.log 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26499.log.gz 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27072.log 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34703.log.gz 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34244.log 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36813.log.gz 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25942.log 2024-09-16T13:20:35.549 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27072.log.gz 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.34244.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27527.log 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34244.log.gz 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35488.log 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.25942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25942.log.gz 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30249.log 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27527.log.gz 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30513.log 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35488.log.gz 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25827.log 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30249.log.gz 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26267.log 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30513.log.gz 2024-09-16T13:20:35.550 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 /var/log/ceph/ceph-client.admin.25827.log: --verbose -- /var/log/ceph/ceph-client.admin.36161.log 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25827.log.gz 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34916.log 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26267.log.gz 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34967.log 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36161.log.gz 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32200.log 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34916.log.gz 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36157.log 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34967.log.gz 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36379.log 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32200.log.gz 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26261.log 2024-09-16T13:20:35.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36157.log.gz 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33435.log 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36379.log.gz 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33408.log 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26261.log.gz 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.33435.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26016.log 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33435.log.gz 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31060.log 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33408.log.gz 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37349.log 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26016.log.gz 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35648.log 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31060.log.gz 2024-09-16T13:20:35.552 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22269.log 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37349.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37349.log.gz 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35589.log 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35648.log.gz 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26931.log 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.22269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22269.log.gz 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36080.log 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35589.log.gz 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28783.log 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26931.log.gz 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33075.log 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36080.log.gz 2024-09-16T13:20:35.553 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30033.log 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28783.log.gz 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28451.log 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33075.log.gz 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30033.log.gz 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30567.log 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34788.log 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28451.log.gz 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28742.log 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30567.log.gz 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27596.log 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34788.log.gz 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.28742.log: -- /var/log/ceph/ceph-client.admin.36799.log 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28742.log.gz 2024-09-16T13:20:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31704.log 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27596.log.gz 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36799.log.gz 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30437.log 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31704.log.gz 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose /var/log/ceph/ceph.log: -- /var/log/ceph/ceph-client.admin.26885.log 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30007.log 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30437.log.gz 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34200.log 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26885.log.gz 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32447.log 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30007.log.gz 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37376.log 2024-09-16T13:20:35.555 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34200.log.gz 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32540.log 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32447.log.gz 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37376.log: 90.8% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37376.log.gz 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27435.log 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32540.log.gz 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-osd.0.log: -5 --verbose -- /var/log/ceph/ceph-client.0.26919.log 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36568.log 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27435.log.gz 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26287.log 2024-09-16T13:20:35.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.0.26919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32799.log 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36568.log.gz 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31744.log 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26287.log: 94.9% 0.0% -- replaced with /var/log/ceph/ceph-client.0.26919.log.gz 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26287.log.gz 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27817.log 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32799.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33788.log 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31744.log: -- replaced with /var/log/ceph/ceph-client.admin.32799.log.gz 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31744.log.gz 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27817.log.gz 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37484.log 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34080.log 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33788.log.gz 2024-09-16T13:20:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37403.log 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37484.log.gz 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33039.log 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34080.log.gz 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28241.log 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37403.log.gz 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33540.log 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33039.log.gz 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30825.log 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28241.log.gz 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26268.log 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33540.log.gz 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37511.log 2024-09-16T13:20:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30825.log.gz 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28555.log 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26268.log.gz 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.37511.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34693.log 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37511.log.gz 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36784.log 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28555.log.gz 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36107.log 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34693.log.gz 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32697.log 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36784.log.gz 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26500.log 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26063.log 2024-09-16T13:20:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36107.log.gz 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32697.log.gz 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29061.log 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26500.log.gz 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31511.log 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26063.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34129.log 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29061.log: -- replaced with /var/log/ceph/ceph-client.admin.26063.log.gz 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29061.log.gz 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37079.log 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31511.log.gz 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26691.log 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34129.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32889.log 2024-09-16T13:20:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37079.log: -- replaced with /var/log/ceph/ceph-client.admin.34129.log.gz 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37079.log.gz 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36583.log 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26691.log.gz 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32889.log.gz 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35675.log 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36583.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36583.log.gz 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26503.log 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35675.log.gz 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26071.log 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35344.log 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26503.log.gz 2024-09-16T13:20:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26071.log.gz 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27206.log 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36357.log 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35344.log.gz 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27206.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34829.log 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27206.log.gz 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36357.log.gz 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32502.log 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27404.log 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34829.log.gz/var/log/ceph/ceph-client.admin.32502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22411.log 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32502.log.gz 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr:gzip /var/log/ceph/ceph-client.admin.27404.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36158.log 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27404.log.gz 2024-09-16T13:20:35.562 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.22411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22411.log.gz 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37160.log 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32951.log 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36158.log.gz 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37160.log.gz 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36591.log 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32951.log.gz 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36594.log 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32633.log 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36591.log.gz 2024-09-16T13:20:35.563 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36594.log.gz 2024-09-16T13:20:35.564 INFO:teuthology.orchestra.run.smithi151.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T13:20:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28622.log 2024-09-16T13:20:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32356.log 2024-09-16T13:20:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32633.log.gz 2024-09-16T13:20:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.28622.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28622.log.gz-5 2024-09-16T13:20:35.567 INFO:teuthology.orchestra.run.smithi098.stderr:--verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32356.log.gz 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37646.log 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26487.log 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.37646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37646.log.gz 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35837.log 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26487.log.gz 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27114.log 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35837.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35837.log.gz 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26069.log 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.27114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27114.log.gz 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35211.log 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.26069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25890.log 2024-09-16T13:20:35.568 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26069.log.gz 2024-09-16T13:20:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35211.log.gz 2024-09-16T13:20:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.25890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25890.log.gz 2024-09-16T13:20:35.630 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T13:20:35.634 INFO:teuthology.orchestra.run.smithi151.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T13:20:35.778 INFO:teuthology.orchestra.run.smithi098.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T13:20:35.780 INFO:teuthology.orchestra.run.smithi098.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T13:20:36.369 INFO:teuthology.orchestra.run.smithi151.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T13:20:36.564 INFO:teuthology.orchestra.run.smithi151.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T13:20:36.810 INFO:teuthology.orchestra.run.smithi098.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T13:20:37.028 INFO:teuthology.orchestra.run.smithi151.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T13:20:37.296 INFO:teuthology.orchestra.run.smithi098.stderr: 95.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T13:20:37.610 INFO:teuthology.orchestra.run.smithi098.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T13:20:37.612 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T13:20:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m2.205s 2024-09-16T13:20:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m5.918s 2024-09-16T13:20:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.284s 2024-09-16T13:20:38.895 INFO:teuthology.orchestra.run.smithi151.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T13:20:38.897 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-09-16T13:20:38.897 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m3.447s 2024-09-16T13:20:38.897 INFO:teuthology.orchestra.run.smithi151.stderr:user 0m6.991s 2024-09-16T13:20:38.897 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m0.256s 2024-09-16T13:20:38.898 INFO:tasks.ceph:Archiving logs... 2024-09-16T13:20:38.898 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295/remote/smithi098/log 2024-09-16T13:20:38.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T13:20:39.866 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295/remote/smithi151/log 2024-09-16T13:20:39.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T13:20:40.840 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T13:20:40.849 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T13:20:40.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T13:20:40.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T13:20:40.937 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T13:20:40.946 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T13:20:40.946 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T13:20:40.948 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T13:20:40.969 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T13:20:40.969 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2024-09-16T13:20:40.969 INFO:teuthology.orchestra.run.smithi098.stdout:+hv01.front.sepi 67.205.162.81 3 u 11 64 177 0.122 0.724 1.197 2024-09-16T13:20:40.970 INFO:teuthology.orchestra.run.smithi098.stdout:+hv02.front.sepi 63.231.80.2 3 u 11 64 177 0.112 -0.105 1.171 2024-09-16T13:20:40.970 INFO:teuthology.orchestra.run.smithi098.stdout:*hv03.front.sepi 74.6.168.72 3 u 10 64 177 0.127 0.321 1.177 2024-09-16T13:20:40.970 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T13:20:40.971 INFO:teuthology.orchestra.run.smithi151.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T13:20:40.971 INFO:teuthology.orchestra.run.smithi151.stdout:============================================================================== 2024-09-16T13:20:40.971 INFO:teuthology.orchestra.run.smithi151.stdout:+hv01.front.sepi 67.205.162.81 3 u 13 64 177 0.050 1.167 0.746 2024-09-16T13:20:40.972 INFO:teuthology.orchestra.run.smithi151.stdout:+hv02.front.sepi 63.231.80.2 3 u 12 64 177 0.093 1.787 1.475 2024-09-16T13:20:40.972 INFO:teuthology.orchestra.run.smithi151.stdout:*hv03.front.sepi 74.6.168.72 3 u 13 64 177 0.104 0.464 0.903 2024-09-16T13:20:40.972 INFO:teuthology.orchestra.run.smithi151.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T13:20:40.972 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T13:20:40.990 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T13:20:40.990 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T13:20:40.999 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T13:20:41.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T13:20:41.017 INFO:teuthology.task.internal:Duration was 748.853796 seconds 2024-09-16T13:20:41.018 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T13:20:41.026 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T13:20:41.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T13:20:41.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T13:20:41.082 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T13:20:41.083 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T13:20:41.083 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T13:20:41.094 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-09-16T13:20:41.095 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T13:20:41.107 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T13:20:41.107 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T13:20:41.109 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T13:20:41.146 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T13:20:41.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T13:20:41.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T13:20:41.289 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T13:20:41.299 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T13:20:41.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T13:20:41.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T13:20:41.330 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T13:20:41.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:20:41.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:20:41.353 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-09-16T13:20:41.387 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-09-16T13:20:41.421 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:20:41.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:20:41.427 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T13:20:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T13:20:41.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T13:20:41.442 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T13:20:41.443 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295/remote/smithi098 2024-09-16T13:20:41.443 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T13:20:41.476 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907295/remote/smithi151 2024-09-16T13:20:41.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T13:20:41.506 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T13:20:41.506 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T13:20:41.511 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T13:20:41.517 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T13:20:41.529 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T13:20:41.529 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T13:20:41.564 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T13:20:41.565 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T13:20:41.567 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T13:20:41.572 INFO:teuthology.orchestra.run.smithi098.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 13:20 /home/ubuntu/cephtest 2024-09-16T13:20:41.574 INFO:teuthology.orchestra.run.smithi151.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 13:20 /home/ubuntu/cephtest 2024-09-16T13:20:41.577 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T13:20:41.586 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T13:20:41.608 INFO:teuthology.run:Summary data: description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_cache_writeback} duration: 748.8537964820862 flavor: default owner: scheduled_sangadi@teuthology success: true 2024-09-16T13:20:41.608 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T13:20:41.690 INFO:teuthology.run:pass